Skip to content

Commit 30dcde1

Browse files
author
Elliot Boschwitz
authored
Created automation for Packages.Microsoft publishing (#431)
* Dockerfile which creates container for publishing * publish.sh script uploads to either testing or prod * readme with step-by-step docs
1 parent 3ac165e commit 30dcde1

File tree

6 files changed

+283
-0
lines changed

6 files changed

+283
-0
lines changed

‎.bumpversion.cfg‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,4 @@ serialize =
1010

1111
[bumpversion:file:build_scripts/rpm/mssql-cli.spec]
1212

13+
[bumpversion:file:release_scripts/Packages.Microsoft/publish.sh]

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,3 +120,7 @@ env*/
120120

121121
# .env file
122122
.env
123+
124+
# cert and config files for Packages.Microsoft publishing
125+
release_scripts/Packages.Microsoft/private.pem
126+
release_scripts/Packages.Microsoft/config.json

‎release_scripts/Packages.Microsoft/Dockerfile‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
FROM amd64/ubuntu:18.04 AS builder
2+
3+
RUN apt-get update
4+
RUN apt-get -y install wget curl nano sudo gnupg gnupg2 gnupg1 jq
5+
RUN apt-get -y install software-properties-common
6+
RUN apt-get -y install apt-transport-https
7+
8+
# Requirements for installing the Repo CLI for Packages.Microsoft
9+
ADD ./release_scripts/Packages.Microsoft/config.json /root/.repoclient/config.json
10+
ADD ./release_scripts/Packages.Microsoft/private.pem /root/private.pem
11+
12+
# Install Repo CLI requirements
13+
RUN curl http://tux-devrepo.corp.microsoft.com/keys/tux-devrepo.asc > tux-devrepo.asc
14+
RUN apt-key add tux-devrepo.asc
15+
RUN echo "deb [arch=amd64] http://tux-devrepo.corp.microsoft.com/repos/tux-dev/ xenial main" | tee /etc/apt/sources.list.d/tuxdev.list
16+
RUN apt-get update
17+
18+
# Add mssql-cli repo
19+
WORKDIR /root
20+
RUN mkdir Repos
21+
RUN mkdir Repos/mssql-cli
22+
ADD . Repos/mssql-cli
23+
24+
# add privileges to publish script
25+
WORKDIR /root/Repos/mssql-cli
26+
RUN chmod +x release_scripts/Packages.Microsoft/publish.sh

‎release_scripts/Packages.Microsoft/README.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Publishing to Packages.Microsoft
2+
> Please ensure that stable versions of rpm and deb have already been published to daily storage.
3+
4+
> Publishing to the Microsoft Repo may only be completed on corp-net. Support for non-corp-net access is not available at the time of this writing. Repo CLI docs may be accessed [here](http://csd-linux-publishing-service.azurewebsites.net/).
5+
6+
The following steps are required to deploy Linux packages to Packages.Microsoft:
7+
1. Build a Linux environment which supports the Repo CLI installation.
8+
2. Install [Repo CLI](http://csd-linux-publishing-service.azurewebsites.net/client#commands), a tool used to publish to Packages.Microsoft.
9+
3. Publish to each supported Linux distribution.
10+
11+
Much of this process has been automated, however, each step may be conducted manually if desired.
12+
13+
## Creating Linux Environment
14+
A Linux environment needs to be created to install [Repo CLI](http://csd-linux-publishing-service.azurewebsites.net/client#commands), which requires:
15+
1. AMD architecture.
16+
2. Connection to corp-net.
17+
3. Include both config and certificate files, detailed below.
18+
19+
### Required Files for Repo CLI
20+
[Repo CLI](http://csd-linux-publishing-service.azurewebsites.net/client#commands) is the required API to publish Linux packages to Packages.Microsoft. Two files are required to use this tool:
21+
1. **A certificate** used for authenticating with the tool.
22+
2. **A JSON config file** as listed on the [Repo CLI docs](http://csd-linux-publishing-service.azurewebsites.net/client#commands).
23+
24+
#### Certificate
25+
The certificate is hosted on our team's Key Vault account in Azure, under 'Certificates'. Download the certificate as a PEM file and include it in this directory (Packages.Microsoft). Rename the file as **private.pem**.
26+
27+
* If you plan to use the automated publishing process, place **private.pem** in this directory (`<repo root>/release_scripts/Packages.Microsoft/private.pem`).
28+
* If a custom Linux environment is used, the location needs to be in `/root/private.pem`.
29+
30+
#### Config
31+
An incomplete `config.json` file is located in this directory. Complete the empty attributes as follows:
32+
* `"AADClientId"`: points to the mssql-cli app registration, titled **mssql-cli (Registered with Packages.Microsoft)**. The ID can be obtained by accessing the Azure Portal.
33+
* `"AADResource"` and `"AADTenant"`: both point to Packages.Microsoft-specific resources. To obtain these values, access the [Repo CLI](http://csd-linux-publishing-service.azurewebsites.net/client) home page, click the **Configure the Repo CLI** tab, followed by the **Packages.Microsoft** sub-tab.
34+
35+
If you plan to use the automated publishing process, keep this file located as is. Otherwise, copy this file to `/root/.repoclient/config.json` on your Linux machine.
36+
37+
### Create Linux Container
38+
This directory contains a dockerfile that generates a Linux environment that's compliant with Repo CLI.
39+
40+
If a non-Docker Linux environment is desired, you may skip to [Repo CLI Installation](#repo-cli-installation). However, please ensure your Linux machine:
41+
1. Uses AMD architecture
42+
2. Is connected to corp-net
43+
44+
#### Building Docker Container
45+
> Docker Desktop is required and may be installed for macOS or Windows [here](https://www.docker.com/products/docker-desktop).
46+
47+
Complete the below command below to create a Docker container with dependencies installed for publishing.
48+
49+
From the root repo directory, run:
50+
```sh
51+
docker build -t mssqlcli-publish-msftrepo -f release_scripts/Packages.Microsoft/Dockerfile . --no-cache
52+
```
53+
54+
#### Running Docker Container
55+
This will run the Docker container that was built using the previous command.
56+
```sh
57+
docker run -it mssqlcli-publish-msftrepo bash
58+
```
59+
60+
You may now complete the Repo CLI installation, detailed in the next section.
61+
62+
## Repo CLI Installation
63+
[Repo CLI](http://csd-linux-publishing-service.azurewebsites.net/client) is the the tool used to publish packages to Packages.Microsoft. Install the Repo CLI in your Linux environment as follows:
64+
65+
```sh
66+
sudo apt-get -y install azure-repoapi-client
67+
```
68+
69+
Respond to all prompts to complete installation.
70+
71+
## Publishing to Packages.Microsoft Using Repo CLI
72+
The `publish.sh` script will upload deb and rpm packages to Packages.Microsoft by consulting the `supported_repos_prod.json` or `supported_repos_testing.json` file. This data is used to find supported repositories located in Packages.Microsoft.
73+
74+
`publish.sh` takes `'testing'` or `'prod'` parameters as its second argument. Uploading to the testing distribution is effective for testing the installation and usage of mssql-cli on various Linux systems without affecting production.
75+
76+
### Publishing to Testing Channel
77+
From the mssql-cli folder root, make the following command to upload packages to all repositories in the `supported_repos_testing.json` file:
78+
```sh
79+
# default call will display commands rather than upload packages
80+
release_scripts/Packages.Microsoft/publish.sh $(pwd) 'testing'
81+
82+
# use --upload to publish to Packages.Microsoft
83+
release_scripts/Packages.Microsoft/publish.sh $(pwd) 'testing' --upload
84+
```
85+
86+
Visit the [testing](#testing-distribution-downloads) section for more information on downloading builds from the testing distribution.
87+
88+
### Publishing to Production Channel
89+
When ready for production, change `'testing'` with `'prod'`, which will upload to all repositories in the `supported_repos_prod.json` file:
90+
```sh
91+
# default call will display commands rather than upload packages
92+
release_scripts/Packages.Microsoft/publish.sh $(pwd) 'prod'
93+
94+
# use --upload to publish to Packages.Microsoft
95+
release_scripts/Packages.Microsoft/publish.sh $(pwd) 'prod' --upload
96+
```
97+
98+
## Repo CLI Commands
99+
This list below contains frequently used commands to navigate the Repo CLI API:
100+
101+
#### Upload Single Package
102+
Packages.Microsoft uses **repositories** to distinguish publishing channels. mssql-cli publishes .deb and .rpm packages directly to repositories, which map to a Linux distribution (i.e. Ubuntu 16.04) along with it's distribution channel (i.e. prod or testing). The [list uploaded pacakges](#list-uploaded-packages) command will display a list respositories with previously-published mssql-cli packages.
103+
104+
The call below uploads a .deb or .rpm package to a specificed repository in Packages.Microsoft:
105+
```sh
106+
repoclient package add <package filepath> -r <repo ID>
107+
```
108+
109+
#### List Uploaded Packages
110+
Use the call below to display a history of previously-published mssql-cli packages:
111+
```sh
112+
repoclient package list | jq '.[] | select(.name=="mssql-cli")'
113+
```
114+
115+
#### List Repositories by Linux Version and Distribution Channel
116+
Use the below command to list Packages.Microsoft repository information, with filters for Linux OS and distribution channel:
117+
118+
```sh
119+
repoclient repo list | jq '.[] | select(.url=="<Linux OS name>" and .distribution=="<distribution>")'
120+
121+
# For example:
122+
repoclient repo list | jq '.[] | select(.url=="microsoft-ubuntu-xenial-prod" and .distribution=="testing")'
123+
```

‎release_scripts/Packages.Microsoft/config.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"server": "package-repo-service.corp.microsoft.com",
3+
"port": "443",
4+
"AADClientId": "",
5+
"AADClientCertificate": "/root/private.pem",
6+
"AADResource": "",
7+
"AADTenant": "",
8+
"AADAuthorityUrl": "https://login.microsoftonline.com",
9+
"repositoryId": ""
10+
}

‎release_scripts/Packages.Microsoft/publish.sh‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# Publishes deb and rpm packages to Packages.Microsoft repo
2+
3+
# Validate initial argument is used
4+
if [[ -z "$1" ]]
5+
then
6+
echo "First argument should be path to local repo."
7+
exit 1
8+
fi
9+
10+
# Validate second argument specifices either prod or testing for publishing channel.
11+
# Each dictionary specifies key-value paris for repo URLs with distribution names.
12+
# Both URL and distribution are needed to query for a repo ID, which is later used
13+
# for publishing.
14+
if [[ ${2,,} = 'prod' ]]; then
15+
declare -A repos=( \
16+
["microsoft-ubuntu-xenial-prod"]="xenial" \
17+
["microsoft-ubuntu-bionic-prod"]="bionic" \
18+
["microsoft-debian-jessie-prod"]="jessie" \
19+
["microsoft-debian-stretch-prod"]="stretch" \
20+
["microsoft-rhel7.0-prod"]="trusty" \
21+
["microsoft-rhel7.1-prod"]="trusty" \
22+
["microsoft-rhel7.2-prod"]="trusty" \
23+
["microsoft-rhel7.3-prod"]="trusty" \
24+
["microsoft-rhel7.4-prod"]="trusty" \
25+
["microsoft-centos7-prod"]="trusty" \
26+
["microsoft-centos8-prod"]="centos" \
27+
)
28+
elif [[ ${2,,} = 'testing' ]]; then
29+
declare -A repos=( \
30+
["microsoft-ubuntu-trusty-prod"]="testing" \
31+
["microsoft-ubuntu-xenial-prod"]="testing" \
32+
["microsoft-debian-jessie-prod"]="testing" \
33+
["microsoft-debian-stretch-prod"]="testing" \
34+
["microsoft-opensuse42.3-testing-prod"]="testing" \
35+
["microsoft-rhel7.0-testing-prod"]="testing" \
36+
["microsoft-rhel7.1-testing-prod"]="testing" \
37+
["microsoft-rhel7.3-testing-prod"]="testing" \
38+
["microsoft-rhel7.2-testing-prod"]="testing" \
39+
["microsoft-rhel7.4-testing-prod"]="testing" \
40+
["microsoft-rhel8.0-testing-prod"]="testing" \
41+
["microsoft-centos7-testing-prod"]="testing" \
42+
["microsoft-centos8-testing-prod"]="testing" \
43+
["microsoft-opensuse42.2-testing-prod"]="testing" \
44+
["microsoft-sles12-testing-prod"]="testing" \
45+
["microsoft-ubuntu-bionic-prod"]="testing" \
46+
["microsoft-ubuntu-cosmic-prod"]="testing" \
47+
["microsoft-ubuntu-disco-prod"]="testing" \
48+
["microsoft-debian-buster-prod"]="testing" \
49+
["microsoft-debian-jessie-prod"]="testing" \
50+
["microsoft-debian-stretch-prod"]="testing" \
51+
)
52+
else
53+
echo "Second argument should specify 'prod' or 'testing' for repository distribution type."
54+
exit 1
55+
fi
56+
57+
# Confirm if third optional '--upload' argument is used
58+
if [[ ${3,,} = '--upload' ]]; then
59+
is_upload='True'
60+
61+
# download latest stable deb and rpm packages
62+
wget https://mssqlcli.blob.core.windows.net/daily/deb/mssql-cli_1.0.0-1_all.deb --directory-prefix=/root/
63+
wget https://mssqlcli.blob.core.windows.net/daily/rpm/mssql-cli-1.0.0-1.el7.x86_64.rpm --directory-prefix=/root/
64+
else
65+
is_upload='False'
66+
fi
67+
68+
local_repo=$1
69+
deb_pkg=/root/mssql-cli_1.0.0-1_all.deb
70+
rpm_pkg=/root/mssql-cli-1.0.0-1.el7.x86_64.rpm
71+
72+
# build url_match_string to get repo ID's from above URL names
73+
url_match_str=""
74+
for repo_url in ${!repos[@]}; do
75+
# get key from url string
76+
distribution="${repos[$repo_url]}"
77+
78+
if [[ $url_match_str == "" ]]; then
79+
# only append 'or' to string if not first index
80+
url_match_str="(.url==\"${repo_url}\" and .distribution==\"${distribution}\")"
81+
else
82+
url_match_str="${url_match_str} or (.url==\"${repo_url}\" and .distribution==\"${distribution}\")"
83+
fi
84+
done
85+
86+
# construct string for select statement in jq command,
87+
# filters by repo URL and distribution type
88+
select_stmnt="select(${url_match_str})"
89+
90+
# query for list of IDs from repo urls
91+
list_repo_id=$(repoclient repo list | jq -r ".[] | ${select_stmnt} | @base64")
92+
for repo_data in $(echo "${list_repo_id}"); do
93+
_jq() {
94+
# decode JSON
95+
echo ${repo_data} | base64 --decode | jq -r ${1}
96+
}
97+
repo_id=$(_jq '.id')
98+
repo_type=$(_jq '.type')
99+
100+
# publish deb or rpm package
101+
# '-r' specifies the destination repository (by ID)
102+
# 'break' exits loop if something failed with command
103+
if [[ $repo_type == "apt" ]]; then
104+
command="repoclient package add $deb_pkg -r $repo_id"
105+
elif [[ $repo_type == "yum" ]]; then
106+
command="repoclient package add $rpm_pkg -r $repo_id"
107+
else
108+
echo "No package published for $(_jq '.url')"
109+
break
110+
fi
111+
112+
echo $command
113+
if [[ $is_upload == "True" ]]; then
114+
# publish package
115+
echo "Publishing $repo_type for $repo_url..."
116+
eval "$command || break"
117+
printf "\n"
118+
fi
119+
done

0 commit comments

Comments
 (0)