Portainer Release Testing

Before a release is initiated, the entire Portainer team dedicate time to complete comprehensive testing of the portainerci/portainer:develop & portainerci/agent:develop images.

Due to the complexity of the codebases, merged pull requests have the potential to interact with each other and cause bugs. If a bug is found, it is fixed, pushed and re-tested to ensure that the release is bug free.

So for those that are interested, we have decided to share the testing we complete before each release. As you can see, its an extensive list, however, whilst its extensive, it doesn't (and can never) cover every possible deployment scenario, so we still rely on our community to help whenever bugs are detected.

Test Area Test Expected Result
Deployment In an existing instance of Portainer, ensure a stack is deployed, and that an extension is licensed. Upgrade the existing Portainer Instance with new version Database Upgrade completes, Portainer instance starts, existing settings retained, stack is still manageable, extension remains licensed.
Deploy a new instance of Portainer bound to the docker socket New instance starts as "local" and can navigate around the endpoint
Deploy a new instance of Portainer using the agent New Instance starts with the Agent, and endpoint is up, snapshot completes successfully, and can navigate around the endpoint
Stacks Function In the stacks page, deploy a stack file that comprises two services (eg Moodle or docker blue/green demo stack) Stack Deploys, services created and scaled correctly
Edit the stack to scale one of the services and then update the stack Service that was changes scales up accordingly
In the stack page, create a new stack based on a stackfile on your compute (upload to Portainer) Stack Deploys, services created and scaled correctly
In the stacks page, create a new stack from a git location using git authentication Stack Deploys, services created and scaled correctly
Services Function In the services page, create a new simple nginx service with scheduling mode of global, expose port 80 nginx service gets deployed and an instance running on each node in the cluster
In the services page, create a new simple nginx service with 1 replica and expose port 80 nginx service gets deployed as a single instance
Scale the service to 5 service scales to 5 nodes, evenly distributed across the cluster
In the services page, create a new simple nginx service with scheduling mode of global, attach a placement constrain of node.role==manager nginx service gets deployed and an instance is running on each manager node in the cluster
In the secrets page, create a secret Secret is saved
In the configs page, create a config Config is saved
In the services page, create a new simple nginx service, with 1 replica, attach the config and override the target path to be /config, attach the secret and override the target path to be /secret. nginx starts, console into container and  config and secret file exists
In the services page, create a new simple nginx service, with 1 replica, create a hosts entry of test.net:127.0.0.1 nginx starts, console into container and can ping test.net with 127.0.0.1 responding
Select an existing service, then click update service and pull new image Service is redeployed
Select an existing service, add a new hosts file entry or test.net:127.0.0.1, then save the service Service is redeployed, console in to container and can ping test.net with 12.7.0.0.1
Select an existing service, click on service logs Service logs are displayed showing aggregate logs from all containers underpinning service
Expand an existing service to show the tasks, click the "stats" quick launch icon to show the performance of that container CPU/RAM/Network and Process information is displayed
Expand an existing service to show the tasks, click the "logs" quick launch icon to show the logs of that container Logs of that container are displayed
Containers Function Create a new simple nginx container, expose port 80 Container is deployed on the selected node
Select the created container, click duplicate/edit, expose port 443 Container is recreated and now 80/443 exposed
select the created container, click recreate, pull latest image Container is recreated and all settings are the same
Select the created container, click on console quick launch and start a console session Container console opens and you can enter text into the console
Select the created container, click on stats quick launch Container performance stats are shown in real time
Select the created container, click the logs quick launch container logs are shown in real time
Images Function In the images page, pull an image from Docker Hub image pulls successfully
In the images page, select a small image and click export image is downloaded as a tar file
In the images page, select to import and image, and chose the image exported previously image is imported into the selected host
In the images page, select to build a new image, enter "from nginx" in the input box and give the image a tag of test:latest Image is built and tagged as test:latest
In the images page, select the test:latest image, and then click on remove image is deleted
Networks Function In the networks page, create a new overlay network, make it attachable, connect two containers to it, and ping between each container by name containers can reach each other by name
In the network page, create a new MACVLAN network, make it attachable, connect a container to it Container gets real IP address and can ping externally
Volumes Function In the volumes page, create a local volume Volume gets created
browse the created volume, and upload a file File gets uploaded into the volume
browse the created volume and rename a file in the volume File gets renamed
browse the created volume and download a file in the volume File gets downloaded to your local computer
browse the created volume and delete a file in the volume File gets deleted
Swarm Function On the swarm page, check that all nodes in your cluster are visible All nodes are displayed, status is ready and availability is active
select a worker node in the cluster, place it in drain mode the node goes into drain mode, and Portainer continues functioning
select the drained node, and place it back in to active mode the node goes active, and Portainer continues functioning
Click the cluster visualiser all nodes are active and you can see services running across nodes (at the very least, the Portainer agent should be running on all nodes)
Extensions On the extensions page, License each of the extensions Extensions license successfully
Disable each of the extensions Extensions are disabled
License each of the extensions with a license key that has been revoked Extensions do not license - get "invalid license key" toaster
License each of the extensions and then revoke the license in the Portainer backend Extension gets disabled on next use
Users Function On the users page, Create a new user and tag them as an admin (selector switch), login as that user user can login and see all endpoints and all Portainer settings
Create a standard user, login as that user user can login, however they are unable to access any endpoints
Create a new team, add the standard user into the team, grant the team access to an endpoint, login as the user user can login and see the endpoint their team has been granted access to
Create a standard user, allow that user access to an endpoint, login as that user user can login and see the endpoint they have been granted access to
Endpoints Function On the endpoints page, add a new remote endpoint that is running the Portainer agent (make sure to deploy a new instance of the Portainer agent in the remote cluster) new endpoint is created and can be used
On the endpoints page, add a new edge agent endpoint New endpoint is created and you are shown the details for how to deploy the edge agent
On the endpoints page, add a new remote endpoint that is exposing the docker daemon over TCP (unsecured) new endpoint is created and can be used
On the endpoints page, add a new remote endpoint that is exposing the docker daemon over TCP with TLS with client and server verification new endpoint is created and can be used
On the endpoints page, add a new remote endpoint that is exposing the docker daemon over TCP with TLS with client verification only new endpoint is created and can be used
On the endpoints page, add a new remote endpoint that is exposing the docker daemon over TCP with TLS with server verification only new endpoint is created and can be used
On the endpoints page, add a new remote endpoint that is exposing the docker daemon over TCP with TLS with no client/server verification new endpoint is created and can be used
On the endpoints groups page, create a new group and add one endpoint into the group. On the users page, create a new standard user. Back on the endpoints groups page, manage access for the group, and add the standard user account to the list of allowed users. Logout and login as the standard user User can login and has access to the endpoint in the endpoint group
On the endpoints tags page, create a tag. On the endpoints page, select an endpoint and then assign the created tag to it, and update the endpoint. On the Portainer home page, in the search bar, type in the tag name you created the endpoint with that tag will be displayed, and the tag will be visible in the endpoint bar (underneath the information about the endpoint)
Registries Function On the registries page, enable dockerhub authentication, add credentials, and then update. On the images page, pull an image from a secure registry. image pulls successfully
On the registries page, add a new custom registry, and connect to a locally hosted Docker Registry without authentication. On the images page, tag am image with the custom registry, and then push the image to the registry Image is pushed successfully
On the registries page, add a new custom registry, and connect to a locally hosted Docker Registry with authentication. On the images page, tag am image with the custom registry, and then push the image to the registry Image is pushed successfully
On the registries page, add a new Azure registry, and enter details for an Azure Registry with authentication. On the images page, tag am image with the custom registry, and then push the image to the registry Image is pushed successfully
Settings Enable the security settings to disable bind mounts and privileged mode for non-admins. Logout and login as a standard user, try to deploy a container and use a bind mount or select privileged mode bind mount option is not displayed as an option. Privileged mode is not displayed as an option
Enable host management features New sidebar menu option "host jobs" is shown
Authentication
LDAP
For an OpenLDAP based environment, Configure LDAP, with server and reader details, test connectivity. Green tick is displayed
Configure LDAP user and group filters specific for OpenLDAP, configure user auto-provisioning.  Create a group called devs in LDAP server, a user called testldapuser and add the user to the devs group. In Portainer create a team called devs, and allow the devs team to access an endpoint. make sure no user called testldapuser exists in Portainer. Logout and login as testuser. testldapuser can login and have access to the endpoint the devs group was granted access to
For an Active Directory based environment, Configure LDAP, with server and reader details, test connectivity. Green tick is displayed
Configure LDAP user and group filters specific for MS AD, configure user auto-provisioning.  Create a group called devs in LDAP server, a user called testaduser and add the user to the devs group. In Portainer create a team called devs, and allow the devs team to access an endpoint. make sure no user called testaduser exists in Portainer. Logout and login as testuser. testaduser can login and have access to the endpoint the devs group was granted access to
Role-Based Access Control User assigned as endpoint-admin against an endpoint User can log in and see the endpoint they are assigned to, click on the endpoint and has admin abilities against all docker resources in the endpoint
User assigned as helpdesk against an endpoint User can log in and see the endpoint they are assigned to, click on the endpoint and has read-only abilities against all docker resources in the endpoint
User assigned as standard user against an endpoint User can log in and see the endpoint they are assigned to, click on the endpoint and has admin abilities against docker resources in the endpoint that are restricted to them or are public
User assigned as read-only against an endpoint User can log in and see the endpoint they are assigned to, click on the endpoint and has read-only abilities against docker resources in the endpoint that are restricted to them or are public
User in team assigned as endpoint-admin against an endpoint User can log in and see the endpoint they are assigned to, click on the endpoint and has admin abilities against all docker resources in the endpoint
User in team assigned as helpdesk against an endpoint User can log in and see the endpoint they are assigned to, click on the endpoint and has read-only abilities against all docker resources in the endpoint
User in team assigned as standard user against an endpoint User can log in and see the endpoint they are assigned to, click on the endpoint and has admin abilities against docker resources in the endpoint that are restricted to them or are public
User in team assigned as read-only against an endpoint User can log in and see the endpoint they are assigned to, click on the endpoint and has read-only abilities against docker resources in the endpoint that are restricted to them or are public
User assigned as endpoint-admin against an endpoint group User can log in and see the endpoint they are assigned to, click on the endpoint and has admin abilities against all docker resources in the endpoint
User assigned as helpdesk against an endpoint group User can log in and see the endpoint they are assigned to, click on the endpoint and has read-only abilities against all docker resources in the endpoint
User assigned as standard user against an endpoint group User can log in and see the endpoint they are assigned to, click on the endpoint and has admin abilities against docker resources in the endpoint that are restricted to them or are public
User assigned as read-only against an endpoint group User can log in and see the endpoint they are assigned to, click on the endpoint and has read-only abilities against docker resources in the endpoint that are restricted to them or are public
User in team assigned as endpoint-admin against an endpoint group User can log in and see the endpoint they are assigned to, click on the endpoint and has admin abilities against all docker resources in the endpoint
User in team assigned as helpdesk against an endpoint group User can log in and see the endpoint they are assigned to, click on the endpoint and has read-only abilities against all docker resources in the endpoint
User in team assigned as standard user against an endpoint group User can log in and see the endpoint they are assigned to, click on the endpoint and has admin abilities against docker resources in the endpoint that are restricted to them or are public
User in team assigned as read-only against an endpoint group User can log in and see the endpoint they are assigned to, click on the endpoint and has read-only abilities against docker resources in the endpoint that are restricted to them or are public
External Authentication External Authentication with Okta is configured following the Userguide A user that is defined in Okta can sign in to Portainer successfully using Okta
External Authentication with Auth0 is configured following the Userguide A user that is defined in Auth0 can sign in to Portainer successfully using Auth0
External Authentication with Google is configured following the Userguide A user that is defined in Google can sign in to Portainer successfully using Google
External Authentication with GitLab is configured following the Userguide A user that is defined in GitLab can sign in to Portainer successfully using GitLab
External Authentication with GitHub is configured following the Userguide A user that is defined in GitHub can sign in to Portainer successfully using GitHub
External Authentication with Keycloak is configured following the Userguide A user that is defined in Keycloak can sign in to Portainer successfully using Keycloak
External Authentication with Azure is configured following the Userguide A user that is defined in Azure can sign in to Portainer successfully using Azure
Registry Manager Click browse on a locally hosted Docker Registry without authentication All repositories are listed. Inside each repository the Add tag, Retag, remove and repository delete functionality are working as intended
Click browse on a locally hosted Docker Registry with authentication All repositories are listed. Inside each repository the Add tag, Retag, remove and repository delete functionality are working as intended
Click browse on an Azure with authentication All repositories are listed. Inside each repository the Add tag, Retag, remove and repository delete functionality are working as intended

Leave a comment!

All fields marked with an asterisk* are required.