aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorKonstantin Ryabitsev <konstantin@linuxfoundation.org>2018-04-14 11:35:18 -0400
committerSitaram Chamarty <sitaramc@gmail.com>2018-04-16 09:57:58 +0530
commit4191404225e1b86534392f0fdbeb5b766bb0f45d (patch)
tree85ea805b6aff386b6454c158339890d2e0c343e0
parentallow hard tabs in template data (diff)
downloadgitolite-gentoo-4191404225e1b86534392f0fdbeb5b766bb0f45d.tar.gz
gitolite-gentoo-4191404225e1b86534392f0fdbeb5b766bb0f45d.tar.bz2
gitolite-gentoo-4191404225e1b86534392f0fdbeb5b766bb0f45d.zip
Reduce disk writes in post-compile triggers
We have a system with thousands of repositories located on disks where: 1. write operations are much slower than read operations 2. IO is burstable but is throttled after a certain limit In such conditions, writing out 10-15 git config settings per each of the 15,000 repositories resulted in around 200,000 calls to git-config and 200,000 disk writes to 15,000 files on every gitolite-admin push, even if there were no changes to any git configs. With this change, we call git-config once to read the entirety of the git config for the repo, and issue writes only if there are changes to any of the settings. Similarly, we only touch git-daemon-export-ok if such file doesn't already exist to avoid another few thousand unnecessary writes. As a result, our gitolite-admin push times went from 10-12 minutes to 20 seconds. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
-rwxr-xr-xsrc/triggers/post-compile/update-git-configs5
-rwxr-xr-xsrc/triggers/post-compile/update-git-daemon-access-list2
2 files changed, 5 insertions, 2 deletions
diff --git a/src/triggers/post-compile/update-git-configs b/src/triggers/post-compile/update-git-configs
index b048992..6eb2f46 100755
--- a/src/triggers/post-compile/update-git-configs
+++ b/src/triggers/post-compile/update-git-configs
@@ -39,12 +39,15 @@ sub fixup_config {
my $creator = creator($pr);
my $gc = git_config( $pr, '.', 1 );
+ my $ac = `git config --file $RB/$pr.git/config -l`;
while ( my ( $key, $value ) = each( %{$gc} ) ) {
next if $key =~ /^gitolite-options\./;
$value =~ s/(@\w+)/expand_group($1)/ge if $rc{EXPAND_GROUPS_IN_CONFIG};
+ my $lkey = lc $key;
+ next if $ac =~ /^\Q$lkey\E=\Q$value\E$/m;
if ( $value ne "" ) {
system( "git", "config", "--file", "$RB/$pr.git/config", $key, $value );
- } else {
+ } elsif ( $ac =~ /^\Q$lkey\E=/m ) {
system( "git", "config", "--file", "$RB/$pr.git/config", "--unset-all", $key );
}
}
diff --git a/src/triggers/post-compile/update-git-daemon-access-list b/src/triggers/post-compile/update-git-daemon-access-list
index b1cf460..ade97a8 100755
--- a/src/triggers/post-compile/update-git-daemon-access-list
+++ b/src/triggers/post-compile/update-git-daemon-access-list
@@ -24,7 +24,7 @@ for my $d (`$cmd | gitolite access % daemon R any`) {
my @F = split "\t", $d;
if ($F[2] =~ /DENIED/) {
unlink "$RB/$F[0].git/$EO";
- } else {
+ } elsif (! -f "$RB/$F[0].git/$EO") {
textfile( file => $EO, repo => $F[0], text => "" );
}
}