3.3. Write and release a new plugin

Author email	sebastien.weber@cemes.fr
Last update	August 2024
Difficulty	Intermediate

In this tutorial, we will learn how to create a brand new plugin for adding either instruments, models or extensions. We will then release it on PyPI so that every PyMoDAQ user can use it and contribute to its development.

3.3.1. Prerequisites

We suppose that you have followed these tutorials:

In the latter, we presented how to interact with existing repositories but what if:

you have an instrument from a manufacturer that doesn’t have yet its package?
you want to build a brand new extension to the DashBoard?

No worries, you don’t have to start from scratch, but from a fairly complete template package!

3.3.2. The PyMoDAQ’s plugin template repository

Among all the PyMoDAQ related GitHub repositories, there is one that is a bit different. This is the pymodaq_plugins_template (see Fig. 3.3).

../_images/template_repo.png — Fig. 3.3 The Template repository to create new plugin packages!

You see that on this repository home page, a new green button Use this template appeared (red box on figure). By clicking on it, you’ll be prompted to create a new repository. In the next page, you’ll be prompted to enter a owner and a name for the repository (see Fig. 3.4).

../_images/create_new_repo.png — Fig. 3.4 The creation page of the new plugin repository

You can choose as a owner either yourself or an organisation that you are part of.

Note

If you wish to be part of the PyMoDAQ organization, send an email to the mailing list pymodaq@services.cnrs.fr. You can host there your future package and be set as the manager.

The name of the plugin as to follow the naming convention pymodaq_plugins_<my_repo_name> where you have to replace <my_repo_name> by the name of the manufacturer if you’re planning to add instruments or a clear name for your application/extension…

Make the repository Public because we want to share our work within the PyMoDAQ community!

That’s it, our new GitHub repository compatible with PyMoDAQ is created. We now have to configure it properly.

3.3.3. Configuring a new plugin repository

For a correct configuration (for your plugin to be installable and recognised by PyMoDAQ), you’ll have to modify a few files and folders. Fig. 3.5 highlights the package initial structure. You’ll have to:

rename with the new package name the two directories highlighted in red.
fill in the appropriate information in plugin_info.toml and README.rst files, highlighted in green.
rename the python instrument file, highlighted in purple with the dedicated instrument name (see Story of an instrument plugin development for details on instrument, python file and class name convention).
add appropriate default settings in the config_template.toml file (do not rename it) in the resources folder.
remove the unused instrument example files of the template repository in the daq_move_plugins and daq_viewer_plugins subfolders.
Modify and configure the automatic publication of your package on the PyPI server (see Releasing on PyPI).

../_images/template_repo_structure.png — Fig. 3.5 The template package initial structure

3.3.4. Releasing on PyPI

3.3.4.1. What is PyPI? What is TestPyPI?

In the Python ecosystem, we often install packages using the pip application. But what happens when we execute pip install mypackage? Well pip is actually looking on a web server for the existence of such a package, then download and install it. This server is the PyPI Python Package Index.

Developers who wish to share their package with others can therefore upload their package there as it is so easy to install it using pip. In our case, we will upload there our plugin as a Python package.

In the following, we will release our plugin on TestPyPI. It is exactly the same as PyPI, except that the Python packages that are stored there are not accessible with pip. It has been created so that we can safely test the release procedure without interacting with the actual PyPI. When we will be ready to actually release a plugin, we will just have to follow the procedure bellow, replacing TestPyPI by PyPI.

3.3.4.2. Create an account on TestPyPI

Let’s go to test.pypi.org to create an account.

../_images/pypi_register.png — Fig. 3.6 Creation of an account on TestPyPI.

After the registration, we have to configure the two factor authentication (2FA). We first need to generate recovery codes.

../_images/pypi_recovery_codes.png — Fig. 3.7 Generate recovery codes.

It will generate 8 of them. Save the .txt file on a safe drive.

../_images/pypi_save_recovery_codes.png — Fig. 3.8 Save the recovery codes.

To configure 2FA, we will need to scan a QR code with an authentication application. If you don’t have one, you can use the Firefox extension called Authenticator. We will install this one in this tutorial, but if you already have another one (1Password for example) you can use it instead.

../_images/firefox_authenticator.png — Fig. 3.9 Authenticator Firefox extension.

Then, we will add 2FA with an authentication application.

../_images/pypi_authentication_application.png — Fig. 3.10 2FA with an authentication application.

Use Authenticator to scan the QR code. It will give us a 6-digit code that we will enter in the form.

../_images/pypi_qr_code.png — Fig. 3.11 Configure the 2FA application.

Note

If you want to be able to use Authenticator in private browsing mode, think about authorizing the extension for this specific mode, otherwise it will not appear in the extensions bar.

We will finally create an API token. It will be useful in the following to authorize GitHub to connect to our TestPyPI account.

Let’s go to the proper menu.

../_images/pypi_add_api_token.png — Fig. 3.12 Create an API token.

We call this token GitHub account in this example (but you can call it as you want) and make a copy of it.

../_images/pypi_copy_token.png — Fig. 3.13 Copy the token.

Note

Be careful to save the token properly as it will appear only once. If you lose it, you will have to generate a new one.

That’s it for now with TestPyPI. Let’s now configure our GitHub account properly!

3.3.4.3. Release our plugin on TestPyPI with GitHub Actions

We will start by creating a GitHub organization. This is useful if you have several developers working in a team. In the context of experimental physics, it is worth creating an organization for our lab group.

Let’s go in the tab Your organization, choose the free plan, and give it a name.

../_images/create_organization.png — Fig. 3.14 Create an organization.

We will now save the TestPyPI token that we created just before in the settings of the organization, so that it will be authorized to access the TestPyPI account.

Once it is created, go to the Settings tab.

../_images/fk_organization_settings.png — Fig. 3.15 Settings of the organization.

Scroll down the left menu in Security > Secrets and variables > Actions

There we create two organization secrets.

The name of the first one is PYPI_USERNAME and its value is __token__.

The second one is PYPI_PASSWORD, within which we will paste the token from TestPyPI that we created in the previous section.

../_images/fk_organization_new_secret.png — Fig. 3.16 Create new secrets to allow the connection to the TestPyPI account.

Now the organization has the credentials to connect to our TestPyPI account.

Let’s now create a new repository in the organization by using the plugin template, as we did at the beginning of the tutorial.

../_images/plugins_template_create_repository.png — Fig. 3.17 Create a new repository in the organization from the template.

Then clone it on our local machine.

Note

Let’s not forget to change the names of the folders and the files as described in the beginning of the tutorial!

We will now have a look at the .github/workflows folder that is at the root of our repository. There are several files that correspond to GitHub Actions. Those are automated tasks that can be triggered by an action of the user on GitHub. For example, it can trigger some automated tests when someone is pushing some code in his repository. Here we will be particularly interested in the python-publish.yml file.

../_images/plugin_template_configure_github_action.png — Fig. 3.18 The python-publish.yml file.

This file is part of the template, and we do not need to enter into the details of its writing. It basically defines that when we will trigger a release from our GitHub repository, it will upload the current version of the repository to TestPyPI.

We can notice that it makes use of the secrets PYPI_USERNAME and PYPI_PASSWORD that we configured earlier to authenticate to TestPyPI at the moment of the release.

Since here we want to discover the release process by releasing to TestPyPI rather than PyPI, we need to change the last line of the file and replace it by

twine upload -r testpypi dist/*

Note

In the case of a release to the actual PyPI, we should skip this last step!

Finally, we should modify the resources/VERSION file of our repository, so that it corresponds to the release tag that we will use for our first release. We can use 1.0.0.

Commit and push those changes towards the remote repository. We are now ready to try our first release!

On the page of our repository, let’s create a new release.

../_images/github_new_release.png — Fig. 3.19 Create a new release.

We are prompted to a form to describe the release. In particular, we have to define a tag for the release, which should correspond to the resources/VERSION file of the package, we use 1.0.0 as the first tag.

../_images/github_configure_release.png — Fig. 3.20 The release form.

By clicking the Publish release button, we automatically trigger the execution of the GitHub Action that is defined in the python-publish.yml file. It will automatically take care of the upload of the package.

To follow what is going on, we have to go to the Actions tab of our GitHub repository.

../_images/github_action_tab_release_failed.png — Fig. 3.21 The GitHub Actions tab is where we found if the release went according to plan. The red cross indicates that it went wrong.

If we click on the workflow that corresponds to the release, we see that something went wrong during the deploy step.

../_images/github_see_action_log.png — Fig. 3.22 The deploy step of the release action went wrong.

Let’s click on it, it will open the log of the release workflow.

../_images/github_action_log_error.png — Fig. 3.23 Access the log of the workflow to get information about what went wrong. Here it indicates that we used a name for the package that was already taken.

Note

This last step has been done (quite ;) ) on purpose to show how to debug a workflow.

After correcting the name of the package from pymodaq_plugins_fk to pymodaq_plugins_fkk the release process went well!

../_images/github_release_green.png — Fig. 3.24 The workflow went well, we are green!

Let’s make a research of our package on TestPyPI, the upload should be quite instantaneous… Here it is! :)

../_images/pypi_package_published.png — Fig. 3.25 Our package has been uploaded to TestPyPI!! :)

3.3.4.4. What are the consequences of a release on PyPI?

There are several consequences if we release a plugin on the actual PyPI (and not TestPyPI).

First, our newly released plugin will automatically be proposed by the Plugin Manager. How is that miracle possible?! Because we respected the naming convention of our plugin, the Plugin Manager just has to search for the Python packages stored on PyPI that start with pymodaq_plugins_…. It is as simple as that!

Secondly, the list of supported instruments will also be updated.

3.3.5. Resources

If you want to understand better the tools that are used in this tutorials, here are a few external links.

In this GitHub documentation Building and testing Python is explained in details how to write your own GitHub Actions to test and release a Python package.

Here is the PyPI documentation about using GitHub Actions.

Here is the Twine documentation.