d79d043696
Fixes #1039. Rather than opening and closing the Bolt DB instance constantly, the cache now maintains one open `*bolthold.Store` for its lifetime, allowing GC, cache read, and cache write operations to occur concurrently. The major risk is this change is, "is it safe to use one Bolt instance across goroutines concurrently?" [Bolt does document its concurrency requirements](https://github.com/boltdb/bolt?tab=readme-ov-file#transactions), and an analysis of our DB interactions looks to me like it introduces very little risk. Most of the cache operations perform multiple touches to the database; for example `useCache` performs a read to fetch a cache object, and then an update to set its `UsedAt` timestamp. If we wanted to ensure consistency in these operations, they should use a Bolt ReadWrite transaction -- but concurrent access would just be setting the field to the same value anyway. The `gcCache` is the complex operation where a transaction might be warranted -- but doing so would also cause the same bug that #1039 indicates. I believe it is safe to run without a transaction because it is protected by an application-level mutex (to prevent multiple concurrent GCs), it is the only code that performs deletes from the database -- these should guarantee that all its delete attempts are successful. In the event of unexpected failure to do the DB write, `gcCache` deletes from the storage before deleting from the DB, so it should just attempt to cleanup again next run. <!--start release-notes-assistant--> <!--URL:https://code.forgejo.org/forgejo/runner--> - bug fixes - [PR](https://code.forgejo.org/forgejo/runner/pulls/1040): <!--number 1040 --><!--line 0 --><!--description Zml4OiBhbGxvdyBHQyAmIGNhY2hlIG9wZXJhdGlvbnMgdG8gb3BlcmF0ZSBjb25jdXJyZW50bHk=-->fix: allow GC & cache operations to operate concurrently<!--description--> <!--end release-notes-assistant--> Reviewed-on: https://code.forgejo.org/forgejo/runner/pulls/1040 Reviewed-by: earl-warren <earl-warren@noreply.code.forgejo.org> Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net> Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net> |
||
|---|---|---|
| .forgejo | ||
| act | ||
| contrib | ||
| examples | ||
| internal | ||
| release-notes | ||
| testutils | ||
| .dockerignore | ||
| .editorconfig | ||
| .gitattributes | ||
| .gitignore | ||
| .golangci.yml | ||
| .pre-commit-hooks.yaml | ||
| Dockerfile | ||
| go.mod | ||
| go.sum | ||
| LICENSE | ||
| main.go | ||
| Makefile | ||
| README.md | ||
| RELEASE-NOTES.md | ||
| renovate.json | ||
Forgejo Runner
A daemon that connects to a Forgejo instance and runs jobs for continuous integration. The installation and usage instructions are part of the Forgejo documentation.
Reporting security-related issues
Sensitive security-related issues should be reported to security@forgejo.org using encryption.
License
The Forgejo runner is distributed under the terms of the GPL version 3.0 or any later version.
Architectures & OS
The Forgejo runner is supported and tested on amd64 and arm64 (binaries and containers) on Operating Systems based on the Linux kernel.
Work may be in progress for other architectures and you can browse the corresponding issues to figure out how they make progress. If you are interested in helping them move forward, open an issue. The most challenging part is to setup and maintain a native runner long term. Once it is supported by Forgejo, the runner is expected to be available 24/7 which can be challenging. Otherwise debugging any architecture specific problem won't be possible.
Hacking
The Forgejo runner is a dependency of the setup-forgejo action. See the full dependency graph for a global view.
Building
- Install Go and
make(1) make build
Linting
make lint-checkmake lint# will fix some lint errors
Testing
The workflow that runs in the CI uses similar commands.
Without a Forgejo instance
- Install Docker
make test integration-test
The TestRunner_RunEvent test suite contains most integration tests
with real-world workflows and is time-consuming to run. During
development, it is helpful to run a specific test through a targeted
command such as this:
go test -count=1 -run='TestRunner_RunEvent$/local-action-dockerfile$' ./act/runner
With a Forgejo instance
- Run a Forgejo instance locally (for instance at http://0.0.0.0:8080) and create as shared secret
export FORGEJO_RUNNER_SECRET='AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
export FORGEJO_URL=http://0.0.0.0:8080
forgejo forgejo-cli actions register --labels docker --name therunner --secret $FORGEJO_RUNNER_SECRET
make test integration-test# which will run addional tests because FORGEJO_URL is set
end-to-end
- Follow the instructions from the end-to-end tests to run actions tests locally.
./end-to-end.sh actions_teardown# stop the Forgejo and runner daemons running in the end-to-end environment( cd ~/clone-of-the-runner-repo ; make build ; cp forgejo-runner /tmp/forgejo-end-to-end/forgejo-runner )# install the runner built from sources./end-to-end.sh actions_setup 13.0# start Forgejo v13.0 and the runner daemon in the end-to-end environment./end-to-end.sh actions_verify_example echo# run the echo workflowxdg-open http://127.0.0.1:3000/root/example-echo/actions/runs/1# see the logs workflowless /tmp/forgejo-end-to-end/forgejo-runner.log# analyze the runner logsless /tmp/forgejo-end-to-end/forgejo-work-path/log/forgejo.log# analyze the Forgejo logs