oss/forgejo-runner
oss/forgejo-runner
1
0
Fork 0
mirror of https://code.forgejo.org/forgejo/runner.git synced 2025-09-05 18:40:59 +00:00
Code Wiki Activity

fix: artifact cache DB not using indexes for searching (#878)

Browse source 
Uses the `Repo` field as an index during searches of the cache database.  Removes unused indexes.

To measure the performance of this change, I created a synthetic test which wrote 10,000 records into the artifact cache DB.  Of course, all benchmarks are lies that can't be generalized to real-world usage, but it seems clear from the magnitude of improvement that this fixes a flawed implementation, even if it's not perfect.
- Unmodified performance:
    - Write: 196 records/second
    - Read: 1 record/second
- With `Repo` index being used for reads, and other indexes being removed:
    - Write: 347 records/second
    - Read: 22,398 records/second

`Repo` is, I think, the only index that made sense to remain, with an eye on workflow run performance:
- `Key` -- can't be used for index because `findCache` searches for key *prefixes*, not equal values.
- `Version` -- isn't very distinct for different workflow runs (https://code.forgejo.org/actions/cache#cache-version)
- `Complete` - significant portion of the cache DB will be complete, making it the least selective possible index
- `UsedAt` & `CreatedAt` - only used in GC operation, so could remain, but this isn't a performance-sensitive codepath

Closes #874.

<!--start release-notes-assistant-->
<!--URL:https://code.forgejo.org/forgejo/runner-->
- bug fixes
  - [PR](https://code.forgejo.org/forgejo/runner/pulls/878): <!--number 878 --><!--line 0 --><!--description Zml4OiBhcnRpZmFjdCBjYWNoZSBEQiBub3QgdXNpbmcgaW5kZXhlcyBmb3Igc2VhcmNoaW5n-->fix: artifact cache DB not using indexes for searching<!--description-->
<!--end release-notes-assistant-->

Reviewed-on: https://code.forgejo.org/forgejo/runner/pulls/878
Reviewed-by: Michael Kriese <michael.kriese@gmx.de>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
This commit is contained in:
Mathieu Fenniak 2025-08-19 20:19:23 +00:00 • committed by earl-warren
parent 6d2cc80aa1
commit 022d5ad3e7
No known key found for this signature in database
GPG key ID: F128CBE6AB3A7201
2 changed files with 7 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

4
act/artifactcache/handler.go
View file

@ -422,7 +422,7 @@ func findCache(db *bolthold.Store, repo string, keys []string, version string) (
for _, prefix := range keys { for _, prefix := range keys {
// if a key in the list matches exactly, don't return partial matches // if a key in the list matches exactly, don't return partial matches
if err := db.FindOne(cache, if err := db.FindOne(cache,
bolthold.Where("Repo").Eq(repo). bolthold.Where("Repo").Eq(repo).Index("Repo").
And("Key").Eq(prefix). And("Key").Eq(prefix).
And("Version").Eq(version). And("Version").Eq(version).
And("Complete").Eq(true). And("Complete").Eq(true).
@ -438,7 +438,7 @@ func findCache(db *bolthold.Store, repo string, keys []string, version string) (
continue continue
} }
if err := db.FindOne(cache, if err := db.FindOne(cache,
bolthold.Where("Repo").Eq(repo). bolthold.Where("Repo").Eq(repo).Index("Repo").
And("Key").RegExp(re). And("Key").RegExp(re).
And("Version").Eq(version). And("Version").Eq(version).
And("Complete").Eq(true). And("Complete").Eq(true).

10
act/artifactcache/model.go
View file

@ -26,10 +26,10 @@ func (c *Request) ToCache() *Cache {
type Cache struct { type Cache struct {
ID uint64 `json:"id" boltholdKey:"ID"` ID uint64 `json:"id" boltholdKey:"ID"`
Repo string `json:"repo" boltholdIndex:"Repo"` Repo string `json:"repo" boltholdIndex:"Repo"`
Key string `json:"key" boltholdIndex:"Key"` Key string `json:"key"`
Version string `json:"version" boltholdIndex:"Version"` Version string `json:"version"`
Size int64 `json:"cacheSize"` Size int64 `json:"cacheSize"`
Complete bool `json:"complete" boltholdIndex:"Complete"` Complete bool `json:"complete"`
UsedAt int64 `json:"usedAt" boltholdIndex:"UsedAt"` UsedAt int64 `json:"usedAt"`
CreatedAt int64 `json:"createdAt" boltholdIndex:"CreatedAt"` CreatedAt int64 `json:"createdAt"`
} }