centos-git-common

Clone
Source Code
GIT

#12 Created lookaside_upload_sig script based on lookaside_upload
Merged 2 years ago by arrfab. Opened 2 years ago by c4rt0.
c4rt0/centos-git-common sigupload  into  master

file added
+123 
@@ -0,0 +1,123 @@ 
 

+ #!/bin/bash
 

+ 

+ # This script will let you upload sources/blobs to new CentOS lookaside cache
 

+ # requirements:
 

+ #  - curl
 

+ #  - valid TLS certs from https://accounts.centos.org (or dev instance for testing)
 

+ 

+ # Some variables, switch for new url
 

+ lookaside_baseurl=$LOOKASIDE_BASEURL
 

+ hash_parameter="sha512"
 

+ 

+ if [ -z $LOOKASIDE_BASEURL ]; then
 

+ 	lookaside_baseurl="https://git.centos.org"
 

+ 	echo "Base URL set to default: $lookaside_baseurl"
 

+ fi
 

+ 

+ function usage {
 

+ 

+ 	cat <<EOF
 

+ 

+ 	You need to call the script like this : $0 -arguments
 

+ 	

+ 			-f : filename/source to upload (required, default:none)
 

+ 			-n : package name for that source (requred, default:none, example "httpd")
 

+ 			-h : display this help
 

+ 

+ 	It is also possible to amend the default base url (currently set to https://git.centos.org):
 

+ 	LOOKASIDE_BASEURL=<urlOfYourChoice> ./lookaside_upload_sig ...
 

+ EOF
 

+ }
 

+ 

+ function varcheck {
 

+ 	if [ -z "$1" ]; then
 

+ 		usage
 

+ 		exit 1
 

+ 	fi
 

+ 

+ }
 

+ 

+ function f_log {
 

+ 	echo "[+] CentOS Lookaside upload tool -> $*"
 

+ }
 

+ 

+ while getopts “hf:n:” OPTION; do
 

+ 	case $OPTION in
 

+ 	h)
 

+ 		usage
 

+ 		exit 1
 

+ 		;;
 

+ 	f)
 

+ 		file=$OPTARG
 

+ 		;;
 

+ 	n)
 

+ 		pkgname=$OPTARG
 

+ 		;;
 

+ 	?)
 

+ 		usage
 

+ 		exit
 

+ 		;;
 

+ 	esac
 

+ done
 

+ 

+ varcheck $file
 

+ varcheck $pkgname
 

+ 

+ if [ ! -f ~/.centos.cert ]; then
 

+ 	f_log "No mandatory TLS cert found (~/.centos.cert) .."
 

+ 	f_log "please use centos-cert to retrieve your ACO TLS cert"
 

+ 	exit 1
 

+ fi
 

+ 

+ if [ ! -f "${file}" ]; then
 

+ 	f_log "Source to upload ${file} not found"
 

+ 	exit 2
 

+ fi
 

+ 

+ checksum="$(${hash_parameter}sum ${file} | awk '{print $1}')"
 

+ 

+ f_log "Checking if file already uploaded"
 

+ local_size=$(stat -c %s ${file})
 

+ http_code=$(curl -s -o /dev/null -w "%{http_code}" ${lookaside_baseurl}/sources/${pkgname}/${file}/${hash_parameter}/${checksum})
 

+ remote_size=$(curl --silent -i --head ${lookaside_baseurl}/sources/${pkgname}/${file}/${hash_parameter}/${checksum} | grep "Content-Length" | cut -f 2 -d ':' | tr -d [:blank:] | tr -d '\r')
 

+ 

+ if [ "$http_code" -eq 200 ] && [ "$local_size" -eq "$remote_size" ]; then
 

+ 	f_log "File already uploaded"
 

+ 	exit 3
 

+ fi
 

+ 

+ f_log "Initialing new upload to lookaside"
 

+ f_log "URL : $lookaside_baseurl"
 

+ f_log "Source to upload : ${file} "
 

+ f_log "Hash parameter : ${hash_parameter}"
 

+ f_log "Package name: $pkgname"
 

+ f_log "sha1sum: ${checksum}"
 

+ f_log " ====== Trying to upload ======="
 

+ echo ""
 

+ 

+ # Concatenating sha512
 

+ hash_cmd="$(${hash_parameter}sum ${file} | awk '{print $1}')"
 

+ curl ${lookaside_baseurl}/sources/upload_sig.cgi \
 

+ 	--fail \
 

+ 	--cert ~/.centos.cert \
 

+ 	--form "name=${pkgname}" \
 

+ 	--form "hash=${hash_parameter}" \
 

+ 	--form "${hash_parameter}sum=${hash_cmd}" \
 

+ 	--form "file=@${file}" \
 

+ 	--progress-bar | tee /dev/null
 

+ 

+ upload_result="${PIPESTATUS[0]}"
 

+ 

+ if [ "$upload_result" -ne "0" ]; then
 

+ 	f_log "[ERROR] Something didn't work to push to ${lookaside_baseurl}/sources/${pkgname}/${checksum}"
 

+ 	f_log "[ERROR] Verify at the server side"
 

+ 	exit 1
 

+ fi
 

+ 

+ f_log "Validating that source was correctly uploaded ...."
 

+ remote_size=$(curl --silent -i --head ${lookaside_baseurl}/sources/${pkgname}/${file}/${hash}/${checksum} | grep "Content-Length" | cut -f 2 -d ':' | tr -d [:blank:] | tr -d '\r')
 

+ if [ "$local_size" -eq "$remote_size" ]; then
 

+ 	f_log "[SUCCESS] Source should be available at ${lookaside_baseurl}/sources/${pkgname}/${file}/${hash}/${checksum}"
 

+ else
 

+ 	f_log "[ERROR] it seems there is a mismatch with source size and remote file size"
 

+ fi

WIP - do not merge

rebased onto 03f8bc4
2 years ago

You can remove this line since branch parameter is optional.

Replace the "sha1" substring wtih the hash parameter value, if hash was set..

Remove this check in here as well, this parameter is optional.

Add a condition right after this donestatement, the script should fail if both hash and branch are not set.

urls in this section should use pkgname/hash/checksum

Is needs to load the sha algorithm from the hash parameter instead hardcoding the usage of "sha1"

The correct url is ${lookaside_baseurl}/sources/upload_sig.cgi

Change to lookaside_upload since it will not use the new cgi service.

${lookaside_baseurl}/sources/${pkgname}/${hash}/${checksum}

${lookaside_baseurl}/sources/${pkgname}/${hash}/${checksum}

${lookaside_baseurl}/sources/${pkgname}/${hash}/${checksum}

${lookaside_baseurl}/sources/${pkgname}/${hash}/${checksum}

${lookaside_baseurl}/sources/${pkgname}/${hash}/${checksum}

I don't think we need this condition at all, the url is being informed to the user in several places.

1 new commit added

  • Implemented suggested changes to lookaside_upload_sig
2 years ago

Could we use an optional env var for this? for example: use the value of LOOKASIDE_CACHE if it is set otherwise default it to "https://git.centos.org"

This function body needs an indentation level.

shouldn't this use "hash" to run the command and get the value as in the "else" case?

It shouldn't fail if branch is missing.

I am using the following command: ./lookaside_upload_sig -f somefile -n somename -a sha256 and getting the following error: +] CentOS Lookaside upload tool -> [ERROR] Neither branch or hash parameters were specified.

This syntax error is also showing up: ./lookaside_upload_sig: line 156: [: missing]'`.

1 new commit added

  • Modified lookaside_upload_sig file based on received feedback
2 years ago

Since we're not touching lookaside_upload, do we want the new script to support both the old and the new format?

I thought so because we are keeping everything "backwards compatible" and I also assumed all sig related tools would point the "new one", regardless of where the SIG stands - is that still the case?

I thought so because we are keeping everything "backwards compatible" and I also assumed all sig related tools would point the "new one", regardless of where the SIG stands - is that still the case?

I was thinking: old structure == old script (since we're keeping it anyway), new structure == new script

Maybe @bstinson has a preference for this?

Also, I'm not sure we should give people the choice of the hash algorithm used, otherwise we may end up with 1 tarball uploaded twice with 2 different hashes which defeats a bit the idea of that lookaside structure

Which algorithm should we force then? sha1?

we agreed to use sha512 as discussed.

1 new commit added

  • Removed the branch element from the script and forced use of SHA512 algorithm
2 years ago

I bet this line needs to be tweaked

One tiny neat pick, but looks good to me!

1 new commit added

  • Modified comment section
2 years ago

Thanks @pingou, not sure how I missed that.
Edited 2 years ago by c4rt0

LGTM as well.

:thumbsup:

I don't have merge permissions, could anyone please merge this PR if we are happy with it?

As it seems that eveybody is happy and tested it, let me merge it

Pull-Request has been merged by arrfab
2 years ago
Metadata
None
No Tags
Changes Summary 1
+123
file added
lookaside_upload_sig

Powered by Pagure 5.13.3

SSH Hostkey/Fingerprint | Documentation