Snapshot of Chef Supermarket¶
Using the ‘on_demand’ policy on a remote allows to create snapshots of a large repository like the Chef Supermarket effectively. In “on_demand” mode, only the meta-data will be synchronized. Actual cookbooks are not downloaded at sync time, but only when requested from a distribution.
The coobook versions distributed are the ones that were presents during the sync, i.e. this will create a frozen snapshot of the Chef Supermarket. Thanks to the repository versioning feature of Pulp 3, you can create a new repository version (i.e. a new snapshot) by syncing the Chef Supermarket at a later time. Note that this does neither influence existing repository versions nor existing distributions. This new repository version will become visible only after publishing and updating the distribution accordingly.
Create a Remote supermarket¶
Using policy=on_demand for the remote enables “lazy download”: A sync using this remote will download the metadata from the Chef Supermarket (i.e. information about all cookbooks and their respective versions), but no actual cookbook tar files.
Cookbooks will be downloaded only when requested from a distribution. After the first successful download, the cookbooks are stored locally for faster retrieval.
pulp_http POST $BASE_ADDR/pulp/api/v3/remotes/cookbook/cookbook/ name='supermarket' url='https://supermarket.chef.io/' policy=on_demand
export REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/cookbook/cookbook/?name=supermarket | jq -r '.results[0].pulp_href')
Response:
{
"ca_cert": null,
"client_cert": null,
"client_key": null,
"cookbooks": null,
"download_concurrency": 10,
"name": "supermarket",
"password": null,
"policy": "on_demand",
"proxy_url": null,
"pulp_created": "2020-08-31T18:56:18.826267Z",
"pulp_href": "/pulp/api/v3/remotes/cookbook/cookbook/5b43bd86-7c52-425c-a369-5693cdea68c2/",
"pulp_last_updated": "2020-08-31T18:56:18.826301Z",
"tls_validation": true,
"url": "https://supermarket.chef.io/",
"username": null
}
Create a repository supermarket¶
If you don’t already have a repository called supermarket, create one using
the remote just created as its default remote:
pulp_http --body POST $BASE_ADDR/pulp/api/v3/repositories/cookbook/cookbook/ name=supermarket description="Snapshots of Supermarket (using lazy download)" remote=$REMOTE_HREF
export REPO_HREF=$(http $BASE_ADDR/pulp/api/v3/repositories/cookbook/cookbook/?name=supermarket | jq -r '.results[0].pulp_href')
Response:
{
"description": "Snapshots of Supermarket (using lazy download)",
"latest_version_href": "/pulp/api/v3/repositories/cookbook/cookbook/cefae475-f693-4122-9b33-60b883b793c7/versions/0/",
"name": "supermarket",
"pulp_created": "2020-08-31T18:56:19.846680Z",
"pulp_href": "/pulp/api/v3/repositories/cookbook/cookbook/cefae475-f693-4122-9b33-60b883b793c7/",
"remote": "/pulp/api/v3/remotes/cookbook/cookbook/5b43bd86-7c52-425c-a369-5693cdea68c2/",
"versions_href": "/pulp/api/v3/repositories/cookbook/cookbook/cefae475-f693-4122-9b33-60b883b793c7/versions/"
}
Sync repository supermarket¶
Use the repository object to kick off a synchronize task. You are telling pulp
to fetch content from the default remote stored in the repository and mirror the
current content exactly (mirror:=true):
pulp_http POST $BASE_ADDR$REPO_HREF'sync/' mirror:=true
export LATEST_VERSION_HREF=$(http $BASE_ADDR$REPO_HREF | jq -r '.latest_version_href')
Response (finished task status):
{
"child_tasks": [],
"created_resources": [
"/pulp/api/v3/repositories/cookbook/cookbook/cefae475-f693-4122-9b33-60b883b793c7/versions/1/"
],
"error": null,
"finished_at": "2020-08-31T18:59:03.915029Z",
"name": "pulp_cookbook.app.tasks.synchronizing.synchronize",
"parent_task": null,
"progress_reports": [
{
"code": "downloading.metadata",
"done": 1,
"message": "Downloading Metadata",
"state": "completed",
"suffix": null,
"total": 1
},
{
"code": "parsing.metadata",
"done": 25910,
"message": "Parsing Metadata",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "associating.content",
"done": 25910,
"message": "Associating Content",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "unassociating.content",
"done": 0,
"message": "Un-Associating Content",
"state": "completed",
"suffix": null,
"total": null
}
],
"pulp_created": "2020-08-31T18:56:20.899326Z",
"pulp_href": "/pulp/api/v3/tasks/165479ae-a05f-4d2b-bff8-a2500877f9d7/",
"reserved_resources_record": [
"/pulp/api/v3/repositories/cookbook/cookbook/cefae475-f693-4122-9b33-60b883b793c7/",
"/pulp/api/v3/remotes/cookbook/cookbook/5b43bd86-7c52-425c-a369-5693cdea68c2/"
],
"started_at": "2020-08-31T18:56:21.075800Z",
"state": "completed",
"task_group": null,
"worker": "/pulp/api/v3/workers/714038d7-7077-43c5-b980-c9be814dcaa0/"
}
Create a Publication¶
The sync created a new repository version which is reflected in the “latest_version_href” field of the repository.
You can kick off a publish task by specifying the repository version to publish. Alternatively, you can specify repository, which will publish the latest version.
The result of a publish is a publication, which contains all the information
needed for an external package manager like berkshelf to use.
Publications are not consumable until they are hosted by a distribution:
task_result=$(pulp_http POST $BASE_ADDR/pulp/api/v3/publications/cookbook/cookbook/ repository_version=$LATEST_VERSION_HREF)
echo "$task_result"
export PUBLICATION_HREF=$(echo "$task_result" | jq -r '.created_resources[0]')
Response (finished task status):
{
"child_tasks": [],
"created_resources": [
"/pulp/api/v3/publications/cookbook/cookbook/6f6ac342-6fdf-4ad1-8b93-449786093d36/"
],
"error": null,
"finished_at": "2020-08-31T18:59:40.138601Z",
"name": "pulp_cookbook.app.tasks.publishing.publish",
"parent_task": null,
"progress_reports": [
{
"code": "publishing.content",
"done": 25910,
"message": "Publishing Content",
"state": "completed",
"suffix": null,
"total": null
}
],
"pulp_created": "2020-08-31T18:59:05.653566Z",
"pulp_href": "/pulp/api/v3/tasks/992dc136-d38b-4a25-af05-98ca91b4f807/",
"reserved_resources_record": [
"/pulp/api/v3/repositories/cookbook/cookbook/cefae475-f693-4122-9b33-60b883b793c7/"
],
"started_at": "2020-08-31T18:59:05.871864Z",
"state": "completed",
"task_group": null,
"worker": "/pulp/api/v3/workers/714038d7-7077-43c5-b980-c9be814dcaa0/"
}
Create a Distribution at ‘supermarket’ for the Publication¶
To host a publication, you create a distribution which will serve the associated
publication at /pulp_cookbook/content/<distribution.base_path>. In the
example, base_path is set to supermarket.
pulp_http POST $BASE_ADDR/pulp/api/v3/distributions/cookbook/cookbook/ name='supermarket' base_path='supermarket' publication=$PUBLICATION_HREF
export DISTRIBUTION_HREF=$(http $BASE_ADDR/pulp/api/v3/distributions/cookbook/cookbook/?name=supermarket | jq -r '.results[0].pulp_href')
Response (finished task status):
{
"child_tasks": [],
"created_resources": [
"/pulp/api/v3/distributions/cookbook/cookbook/d3dad6b8-817b-42c2-b546-addf72e92bf5/"
],
"error": null,
"finished_at": "2020-08-31T18:59:42.672067Z",
"name": "pulpcore.app.tasks.base.general_create",
"parent_task": null,
"progress_reports": [],
"pulp_created": "2020-08-31T18:59:42.253922Z",
"pulp_href": "/pulp/api/v3/tasks/98dce068-16e5-4006-a1bb-836392335202/",
"reserved_resources_record": [
"/api/v3/distributions/"
],
"started_at": "2020-08-31T18:59:42.522598Z",
"state": "completed",
"task_group": null,
"worker": "/pulp/api/v3/workers/7a1130eb-8483-418f-a370-cecf0e63f60b/"
}
You can have a look at the published “universe” metadata now:
$ http localhost:24816/pulp_cookbook/content/supermarket/universe
In your Berksfile, you can use the following source to access the
Supermarket snapshot:
source 'http://localhost/pulp_cookbook/content/supermarket/'