mirror of
1
Fork 0
forgejo/models
Jason Song a627b885c7
Support compression for Actions logs (#31761)
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015

The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.

That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.

This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.

And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.

`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.

<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">

(cherry picked from commit 33cc5837a655ad544b936d4d040ca36d74092588)

Conflicts:
	assets/go-licenses.json
	go.mod
	go.sum
  resolved with make tidy
2024-08-13 06:51:49 +02:00
..
actions Support compression for Actions logs (#31761) 2024-08-13 06:51:49 +02:00
activities Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
admin Next round of `db.DefaultContext` refactor (#27089) 2023-09-16 14:39:12 +00:00
asymkey [CHORE] Remove SSH DSA tests 2024-08-05 20:34:19 +02:00
auth Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
avatars Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
db Adjust codespell config + make it fix few typos which sneaked in since addition of codespell support (#4857) 2024-08-08 16:07:35 +00:00
dbfs Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
fixtures feat(cli): add `--keep-labels` flag to `forgejo actions register` (#4610) 2024-07-22 07:33:45 +00:00
forgefed enhance test & fix reviews 2024-05-14 08:24:31 +02:00
forgejo/semver Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
forgejo_migrations feat(quota): Humble beginnings of a quota engine 2024-08-02 11:10:34 +02:00
git Rename head branch of pull requests when renaming a branch (#31759) 2024-08-04 18:24:10 +02:00
issues Fix null requested_reviewer from API (#31773) 2024-08-13 06:51:49 +02:00
migrations Clear up old Actions logs (#31735) 2024-08-04 18:24:10 +02:00
organization Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
packages Arch packages implementation (#4785) 2024-08-04 06:16:29 +00:00
perm Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
project Fix the display of project type for deleted projects (#31732) 2024-08-04 10:14:34 +02:00
pull [GITEA] GetScheduledMergeByPullID may involve a system user 2024-02-05 16:09:41 +01:00
quota feat(quota): Humble beginnings of a quota engine 2024-08-02 11:10:34 +02:00
repo Remove unused code from models/repos/release.go (#31756) 2024-08-04 18:24:10 +02:00
secret Clarify Actions resources ownership (#31724) 2024-08-04 18:24:10 +02:00
shared/types Refactor locale&string&template related code (#29165) 2024-02-16 15:20:52 +01:00
system Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
unit Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
unittest Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
user Adjust codespell config + make it fix few typos which sneaked in since addition of codespell support (#4857) 2024-08-08 16:07:35 +00:00
webhook Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
error.go Add merge style `fast-forward-only` (#28954) 2024-02-14 17:19:19 +01:00
fixture_generation.go Replace more db.DefaultContext (#27628) 2023-10-15 17:46:06 +02:00
fixture_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
main_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
org.go Refactor deletion (#28610) 2023-12-25 21:25:29 +01:00
org_team.go Remove GetByBean method because sometimes it's danger when query condition parameter is zero and also introduce new generic methods (#28220) 2023-12-07 15:27:36 +08:00
org_team_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
org_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
repo.go Refactor deletion (#28610) 2023-12-25 21:25:29 +01:00
repo_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
repo_transfer.go chore: use errors.New to replace fmt.Errorf with no parameters will much better (#30621) 2024-04-28 15:39:00 +02:00
repo_transfer_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00