Synchronize a Repository

Users can populate their repositories with content from an external sources by syncing their repository.

Bear in mind that the attached snippets utilize pulp_http. Refer to Workflows to learn more about these utilities and the required tools (httpie and jq).

Create a Remote

In addition to uploading content, pulp_cookbook allows to synchronize a repository with an external content source (that has to provide a “universe” endpoint, like the Chef Supermarket). Creating a remote object informs Pulp about an external content source.

Since the Chef Supermarket is huge, let’s mirror only the pulp, qpid, and ubuntu cookbooks into our repository by setting the cookbooks parameter. First, we have to create a remote:

pulp_http POST $BASE_ADDR/pulp/api/v3/remotes/cookbook/cookbook/ name='foo_remote' url='https://supermarket.chef.io/' policy=immediate cookbooks:='{"pulp": "", "qpid": "", "ubuntu": ""}'
export REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/cookbook/cookbook/?name=foo_remote | jq -r '.results[0].pulp_href')

Response:

{
    "ca_cert": null,
    "client_cert": null,
    "client_key": null,
    "cookbooks": {
        "pulp": "",
        "qpid": "",
        "ubuntu": ""
    },
    "download_concurrency": 10,
    "name": "foo_remote",
    "password": null,
    "policy": "immediate",
    "proxy_url": null,
    "pulp_created": "2020-08-31T18:55:53.000019Z",
    "pulp_href": "/pulp/api/v3/remotes/cookbook/cookbook/4fb67e84-1c87-454a-9cc9-d70f747b78de/",
    "pulp_last_updated": "2020-08-31T18:55:53.000041Z",
    "tls_validation": true,
    "url": "https://supermarket.chef.io/",
    "username": null
}

Create a Repository

If you don’t already have a repository called foo, create one. While it is possible to add the remote as a parameters to every sync operation, it is more convenient to store the default remote in the repository:

pulp_http --body POST $BASE_ADDR/pulp/api/v3/repositories/cookbook/cookbook/ name=foo remote=$REMOTE_HREF
export REPO_HREF=$(http $BASE_ADDR/pulp/api/v3/repositories/cookbook/cookbook/?name=foo | jq -r '.results[0].pulp_href')

Response:

{
    "description": null,
    "latest_version_href": "/pulp/api/v3/repositories/cookbook/cookbook/84be8c38-0889-4135-81c5-3ebb7a490e20/versions/0/",
    "name": "foo",
    "pulp_created": "2020-08-31T18:55:53.983398Z",
    "pulp_href": "/pulp/api/v3/repositories/cookbook/cookbook/84be8c38-0889-4135-81c5-3ebb7a490e20/",
    "remote": "/pulp/api/v3/remotes/cookbook/cookbook/4fb67e84-1c87-454a-9cc9-d70f747b78de/",
    "versions_href": "/pulp/api/v3/repositories/cookbook/cookbook/84be8c38-0889-4135-81c5-3ebb7a490e20/versions/"
}

Sync repository foo

Use the repository object to kick off a synchronize task. You are telling pulp to fetch content from the default remote stored with the repository and add to the latest repository version (mirror:=false):

pulp_http POST $BASE_ADDR$REPO_HREF'sync/' mirror:=false
export LATEST_VERSION_HREF=$(http $BASE_ADDR$REPO_HREF | jq -r '.latest_version_href')

Response (finished task status):

{
    "child_tasks": [],
    "created_resources": [
        "/pulp/api/v3/repositories/cookbook/cookbook/84be8c38-0889-4135-81c5-3ebb7a490e20/versions/1/"
    ],
    "error": null,
    "finished_at": "2020-08-31T18:56:01.346161Z",
    "name": "pulp_cookbook.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "downloading.metadata",
            "done": 1,
            "message": "Downloading Metadata",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        {
            "code": "downloading.artifacts",
            "done": 18,
            "message": "Downloading Artifacts",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 18,
            "message": "Associating Content",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.metadata",
            "done": 18,
            "message": "Parsing Metadata",
            "state": "completed",
            "suffix": null,
            "total": null
        }
    ],
    "pulp_created": "2020-08-31T18:55:54.958181Z",
    "pulp_href": "/pulp/api/v3/tasks/afc496dc-0862-4d9a-8bcc-24225cafbaf7/",
    "reserved_resources_record": [
        "/pulp/api/v3/repositories/cookbook/cookbook/84be8c38-0889-4135-81c5-3ebb7a490e20/",
        "/pulp/api/v3/remotes/cookbook/cookbook/4fb67e84-1c87-454a-9cc9-d70f747b78de/"
    ],
    "started_at": "2020-08-31T18:55:55.156590Z",
    "state": "completed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/7a1130eb-8483-418f-a370-cecf0e63f60b/"
}

When the synchronize task completes successfully, there are two possible outcomes:

1. If the sync changed the content, it creates a new version, which is specified in created_resources. Moreover, the created repository version will become the latest_version of the repository.
2. If the sync did not change the content (i.e. the remote contains no changes compared to the current repository version), no new version is created and created_resources is empty.

You can have a look at the latest repository version:

http "$BASE_ADDR$LATEST_VERSION_HREF"

Response:

{"pulp_href":"/pulp/api/v3/repositories/cookbook/cookbook/84be8c38-0889-4135-81c5-3ebb7a490e20/versions/1/","pulp_created":"2020-08-31T18:55:55.190535Z","number":1,"base_version":null,"content_summary":{"added":{"cookbook.cookbook":{"count":18,"href":"/pulp/api/v3/content/cookbook/cookbooks/?repository_version_added=/pulp/api/v3/repositories/cookbook/cookbook/84be8c38-0889-4135-81c5-3ebb7a490e20/versions/1/"}},"removed":{},"present":{"cookbook.cookbook":{"count":18,"href":"/pulp/api/v3/content/cookbook/cookbooks/?repository_version=/pulp/api/v3/repositories/cookbook/cookbook/84be8c38-0889-4135-81c5-3ebb7a490e20/versions/1/"}}}}