Install and update pyRevit extension from a private repo

Jean-Marc · November 26, 2020, 9:04am

Am I doing something wrong?

long question

I have a private repo _ I know, should be open source right! but it cannot _ where I hosted my custom extension:

extension.json at the root looks like this:

		{
    "builtin": "False",
    "default_enabled": "True",
    "type": "extension",
    "rocket_mode_compatible": "True",
    "name": "B1_ToolBar",
    "description": "Barre d'outils pour BIMOne inc.",
    "author": "Jean-Marc Couffin",
    "author_profile": "https://github.com/jmcouffin",
    "url": "https://github.com/jmcouffin/B1_ToolBar.git",
    "website": "https://bimone.com",
    "image": "",
    "dependencies": []
}

So far so good right.

As it is a private repo I install it like so in command line:
pyrevit extend ui B1_ToolBar https://github.com/jmcouffin/B1_ToolBar.git --dest="C:\pyRevit" --username="xxxxxxx" --password="xxxxxxxx" --debug

and it does install

in debug mode it tells me the pyrevit config.ini file is updated

[B1_ToolBar.extension]
disabled = false
private_repo = false
username = ""
password = ""

but no password or username saved…

Is it expected?
Is there a way to push right away Username and PW?

Because when I want to push an update through the command line

I get the obvious
Error: request failed with status code: 401 (pyRevitLabs.Common.PyRevitException) à pyRevitLabs.Common.GitInstaller.ForcedUpdate(String repoPath, String username, String password) à pyRevitLabs.PyRevit.PyRevitExtensions.UpdateExtension(PyRevitExtension ext, String username, String password) à pyRevitCLI.PyRevitCLI.ProcessArguments() à pyRevitCLI.PyRevitCLI.Main(String[] args)

As is does not have the PW and username stored anywhere. It makes sense
The update command can take the Username and PW

pyrevit extensions update (--all | <extension_name>) [--log=<log_file>] [--username=<username>] [--password=<password>]

as arguments but I want to be able to update through the UI Update button to get my users a seamless process

I tried adding the Username and PW to the config.ini manually (as it is stored in plain text ) but when I launch again
pyrevit extensions update B1_ToolBar I get the same error and I am expecting it to read the credentials stored in the config.ini when launching an update from the CLI

If I put the config.ini back to its initial state (without username and pw store) and I put the username and PW in the extensions in pyrevit like so

I still cannot update it through CLI with pyrevit extensions update B1_ToolBar without the credentials but I can through the update UI button in pyRevit

so

Am I doing something wrong?

and how can I make it do properly? Because I do not get it!

I would like to:

setup my private repo extension in one pyRevit CLI command including credentials and be able to get it updated with the pyRevit UI Update button

Possible? (without powershell or command line tricks because they can cause other troubles)

eprunty · December 4, 2020, 10:01am

I followed your steps above (thank you for outlining so clearly as I had been wondering how to do this!) and swapping –dest=“C:\pyRevit” for –dest="$($ENV:AppData)\pyRevit\extensions" worked for me.

Do you know if it’s possible to create dummy account credentials for access to private repos? I don’t want to leave my GitHub account credentials signed in on someone else’ machine.

Jean-Marc · December 4, 2020, 10:27am

@eprunty creating another github account for all of my people to connect through it worked. I would not leave my personal credentials either. Especially stored in clear in a .ini file

This is a know issue Deploying private extensions (git) using CLI · Issue #801 · eirannejad/pyRevit · GitHub and update extensions from private repo · Issue #964 · eirannejad/pyRevit · GitHub
the implementation for the install from a private repo works but the storage of credentials is lacking functionalities

eprunty · December 4, 2020, 3:21pm

I know you don’t want to use powershell to sort this but I thought this may be useful to others. To install in one go I’ve used the following in powershell (%% used to denote variables in need of updating for anyone looking to do the same):

pyrevit extend ui %extensionName% https://github.com/%extension%.git --dest="$($ENV:AppData)\pyRevit\extensions" --username="%username%" --password="%password%" --debug

#update user config ini file with username and password for private repo
$places = "$($ENV:AppData)\pyRevit"
$lines  = '', '[%extensionName%.extension]', 'disabled = false', 'private_repo = true', 'username = "%username%"', 'password = "%password%"'  
Get-ChildItem $places -Recurse -Include pyRevit_config.ini | ForEach-Object {
  Add-Content $_.FullName -Value $lines
}

The main problem with it is \AppData\Roaming\pyRevit\Extensions gets added to the userextensions key in the pyRevit_config.ini file, meaning that all enabled extensions load twice when Revit opens… Presumably it’s got something to do with pyrevit extend ui in the first line but I can’t figure it out.

Is there a way to do this without adding the standard extensions folder to the user extensions?

(also if it’s run more than once the extension details in the ini file will be duplicated on each run!)

thumDer · December 4, 2020, 10:47pm

Because of this very issue about credentials we chose to go a completely different way. Please tell if you have any objections regarding this workflow.

Instead of pulling our extension to every users machine, we have a copy of the master branch in a network location, that can be accessed by all Revit users, and that network path is added to the default config. (It is read-only to most of the users)
So if I want to distribute the updates to our extension I simply run a PS script on my PC which pulls the repo zip, and unzips it to the network location, overwriting the previous version.

thumDer · December 5, 2020, 12:13am

If anyone is interested here is the ps1 using GitLab API. It’s a bit funky, because in the zip, there is a ‘root’ folder that is named as the commit sha, so not constant. That is why the mess with the $copysourcepath

if (Test-Path $extractto) {
	Get-ChildItem -Path $extractto -Recurse | Remove-Item -Recurse
}

$r = Invoke-RestMethod -Uri $gitlaburi -Method Get -ContentType "application/zip" -Headers @{ 'PRIVATE-TOKEN'='$($gitlabtoken)' } -OutFile $zip
Expand-Archive -LiteralPath $zip -DestinationPath $extractto

$subfoldername =  Get-ChildItem -Path $extractto -Name
$folderpath = Join-Path -Path $extractto -ChildPath $subfoldername
$copysourcepath = Join-Path -Path $folderpath -ChildPath "*"

# clean target folder
if (Test-Path $targetpath) {
	Get-ChildItem -Path $targetpath -Recurse | Remove-Item -Force -Recurse
}

Copy-Item -Path $copysourcepath -Destination $copytargetpath -Recurse -Force

Jean-Marc · December 9, 2020, 1:57pm

I followed your train of thoughts. Works well except it does not work entirely as expected:

the extension is added to the extension manager
the password and username is filled in
but the extension is not considered as installed or enabled

I need either @eirannejad 's workshop or a fix

If I try to install and enable, it won’t let me point the install to the folder listed by the pyrevit extend command as the folder contains the extension data already

eprunty · December 9, 2020, 5:36pm

That’s strange as it installed for both 2019 and 2020 for me. I deleted the existing extension from the folder, removed the extensions’ section from the ini file manually then ran powershell in case that’s the difference? I can’t figure out how to remove the ini section for the extension using powershell, which would be a more elegant and welcome solution… The main application of this will be for machines that it’s not already on.

I’ve added in two more lines of find/replace to remove the user extensions folders (will clear all user extensions out) and to set the check updates on startup option as active.

Pasting again for the whole code with CAPS to highlight required variables:

#pull extension from github
pyrevit extend ui EXTENSIONNAME https://github.com/USER/EXTENSION.git --dest="$($ENV:AppData)\pyRevit\extensions" --username="USERNAME" --password="PASSWORD" #--debug

#remove extensions from user extensions folder
(Get-Content $ENV:AppData\pyRevit\pyRevit_config.ini) | ForEach-Object { $_ -replace 'userextensions =.*','userextensions = []' } | Set-Content $ENV:AppData\pyRevit\pyRevit_config.ini
#activate check updates from github at startup
(Get-Content $ENV:AppData\pyRevit\pyRevit_config.ini) | ForEach-Object { $_ -replace 'checkupdates = false', 'checkupdates = true' } | Set-Content $ENV:AppData\pyRevit\pyRevit_config.ini

#update user config ini file with username and password for private repo
$places = "$($ENV:AppData)\pyRevit"
$lines  = '', '[EXTENSIONNAME.extension]', 'disabled = false', 'private_repo = true', 'username = "USERNAME"', 'password = "PASSWORD"'
Get-ChildItem $places -Recurse -Include pyRevit_config.ini | ForEach-Object {
  Add-Content $_.FullName -Value $lines
}

scartmell · January 7, 2021, 9:21pm

I also would like to go this route. I have a copy of the extension I’ve been developing on a server. Is there a way to use the extend command in pyrevit CLI but pass a folder location instead of a git repo url? Or would this be an entirely powershell command and pyrevit CLI isn’t involved?

eirannejad · January 8, 2021, 6:52am

This is a very interesting thread. I’m going to take a look and list a few things that needs to get fixed with the CLI

Jean-Marc · January 8, 2021, 2:17pm

@eirannejad,
happy to see you step in this one.
a few questions to expand my initial thoughts:

Passing credentials (token as github will deprecate api access through pw)
Is the distinction in terms of process is necessary between custom extensions and builtin?
I see a real ‘value’ in CLI installation of extensions from private repo.
I went down @eprunty line to make it work + a cli to make updates work seamlessly outside of pyrevit itself.

import os os.system('cmd /c "pyrevit extensions update B1_Toolbar --user="username" --password="pw"')

thumDer · January 13, 2021, 10:48am

use pyrevit extensions paths add <extensions_path>

Bmoreawesom3 · February 13, 2021, 7:56pm

This is all way over my head and I am very new to github and how it works. In a nut shell you can take your custom tab folder structure and tools and load it to a github with login and password and pyrevit can connect to it to download and install the custom tab and tools and also update based on changes.

A question is the license file, readme file and the extension file all created by github or are they setup manually?

bare in mind I am new to coding and what different files extension actually do.

Jean-Marc · February 15, 2021, 9:41am

license and readme are automatically created in any github (if you allow it) the extension file is to be created by you.

eprunty · February 18, 2021, 12:40pm

Is there a simple fix to adjust this for token access following GitHub’s password deprecation?

#pull extension from github
pyrevit extend ui EXTENSIONNAME https://github.com/USER/EXTENSION.git --dest="$($ENV:AppData)\pyRevit\extensions" --username="USERNAME" --password="PASSWORD" #--debug

Basic authentication using a password to Git is deprecated and will soon no longer work. Visit Token authentication requirements for Git operations - The GitHub Blog for more information around suggested workarounds and removal dates.

Jean-Marc · February 18, 2021, 1:05pm

have you tried to fill in the token in lieu of the password?
haven’t tried it myself.

Happyhawk · February 26, 2021, 2:21pm

I have tried using an access token instead of a password, and that works fine for installing my extension, but for a Gitea-hosted server and not Github. Nice with a workaround for automatic updating @Jean-Marc, thank you!

eirannejad · February 28, 2021, 1:48am

Okay I figued how to use libgit2sharp library to clone from github with the personal access tokens. Does that solve the problem of deployment without user/pass?

Jean-Marc · February 28, 2021, 2:47pm

It should. Do you have it somewhere I can try? @eirannejad

eirannejad · March 1, 2021, 2:44am

I’ll merge the changes in this week and will keep you posted. It was actually quite and easy add. Just need time to make both the CLI and the pyRevit runtime use the same api