mirror of
1
Fork 0
Commit Graph

97 Commits

Author SHA1 Message Date
kim ec00dbd496
bump gruf/go-ffmpreg to v0.6.3 () 2024-12-16 11:36:01 +00:00
kim 8504043024
bump gruf/go-ffmpreg to v0.6.2 () 2024-12-12 22:02:20 +00:00
kim b84637801a
[chore] update go ffmpreg to v0.6.0 ()
* pull in go-ffmpreg v0.6.0

* add code comment

* grrr linter

* set empty module name when calling ffmpeg / ffprobe
2024-11-06 14:38:13 +01:00
kim 8f288f1689
[bugfix] determine mime-type to use during ffprobe evaluation stage, don't bother checking against file extension ()
* determine mime-type to use during ffprobe evaluation stage, don't bother rechecking by file extension

* set mjpeg content-type

* fix up tests expecting differing default values
2024-11-04 14:58:15 +01:00
kim e86592bc32
[chore] pull in go-ffmpreg v0.4.1 ()
* pull in go-ffmpreg v0.4.1

* bring back GTS_WAZERO_COMPILATION_CACHE
2024-10-28 10:55:48 +00:00
kim bd1866ad8a
update go-ffmpreg to v0.3.1 (pulls in latest wazero too) () 2024-10-06 20:53:03 +00:00
kim c17abea921
update go-structr to v0.8.11 () 2024-10-02 10:58:20 +00:00
kim 095663f5cc
[bugfix] visibility after implicit approval not getting invalidated ()
* replicate issue

* update go-structr to v0.8.10 with internal linked-list fix, small tweaks to caching of interaction requests

* remove debug function

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-09-28 22:47:46 +02:00
kim 2f582e2e33
update gruf / {go-cache, go-maps, go-kv} () 2024-09-27 10:26:50 +00:00
kim 58af95a1d5
[chore] bump go-byteutil v1.2.0 -> v1.3.0 ()
* bump go-byteutil v1.2.0 -> v1.3.0 which has safer (as in long-term API consistency) byte <-> string conversions

* fix test relying on byteutil exported type no longer existing
2024-09-26 19:50:08 +00:00
kim f3e2d36d64
[chore] update go-sched pkg ()
* update go-sched to v1.2.4 which removes some now unused dependencies

* whoops, remove test output
2024-09-26 19:23:41 +00:00
kim 53ee6aef08
[bugfix] s3 media uploaded without content-type ()
* update go-storage dependency, for S3Storage manually call PutObject() so we can set content-type

* update calls to PutFile() to include the contentType
2024-09-26 14:43:10 +02:00
kim 77b095a8c3
[chore] ensure consistent caller name fetching regardless of compiler inlining ()
* move logging levels into log package itself

* ensure inconsistent inlining doesn't mess with log calling function name

* remove unused global variable

* fix log level
2024-09-20 13:30:33 +00:00
kim 84279f6a6a
[performance] cache more database calls, reduce required database calls overall ()
* improvements to caching for lists and relationship to accounts / follows

* fix nil panic in AddToList()

* ensure list related caches are correctly invalidated

* ensure returned ID lists are ordered correctly

* bump go-structr to v0.8.9 (returns early if zero uncached keys to be loaded)

* remove zero checks in uncached key load functions (go-structr now handles this)

* fix issues after rebase on upstream/main

* update the expected return order of CSV exports (since list entries are now down by entry creation date)

* rename some funcs, allow deleting list entries for multiple follow IDs at a time, fix up more tests

* use returning statements on delete to get cache invalidation info

* fixes to recent database delete changes

* fix broken list entries delete sql

* remove unused db function

* update remainder of delete functions to behave in similar way, some other small tweaks

* fix delete user sql, allow returning on err no entries

* uncomment + fix list database tests

* update remaining list tests

* update envparsing test

* add comments to each specific key being invalidated

* add more cache invalidation explanatory comments

* whoops; actually delete poll votes from database in the DeletePollByID() func

* remove added but-commented-out field

* improved comment regarding paging being disabled

* make cache invalidation comments match what's actually happening

* fix up delete query comments to match what is happening

* rename function to read a bit better

* don't use ErrNoEntries on delete when not needed (it's only needed for a RETURNING call)

* update function name in test

* move list exclusivity check to AFTER eligibility check. use log.Panic() instead of panic()

* use the poll_id column in poll_votes for selecting votes in poll ID

* fix function name
2024-09-16 16:46:09 +00:00
kim 6c0bd0e20e
update go-ffmpreg v0.2.5 -> v0.2.6 () 2024-09-10 19:10:19 +00:00
kim 8e5a72ac5c
[performance] ffmpeg ffprobe wrapper improvements ()
* use a single instance of wazero runtime and compiled modules

* remove test output 🤦

* undo process-{media,emoji} changes

* update test runner to include wazero compilation cache

* sign drone.yml

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-23 17:15:35 +02:00
kim 09f24e0446
update go-ffmpreg to v0.2.5 (pulls in latest tetratelabs/wazero) () 2024-08-15 00:08:55 +00:00
kim 6fe96a5611
update go-fastcopy to v1.1.3 () 2024-08-14 16:56:56 +00:00
kim f8d5b9f49c
update go-structr to v0.8.8 () 2024-08-14 12:08:24 +00:00
kim 3045782b49
updates our ffmpreg version, heh () 2024-08-07 15:38:02 +02:00
kim 409b3987ac
update go-ffmpreg to v0.2.3 () 2024-07-20 14:14:58 +01:00
tobi e2ed6ef024
[chore] Go mod tidy / vendor () 2024-07-13 16:23:25 +02:00
kim cde2fb6244
[feature] support processing of (many) more media types ()
* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg

* specify the video codec to use when generating static image from emoji

* update go-storage library (fixes incompatibility after updating go-iotools)

* maintain image aspect ratio when generating a thumbnail for it

* update readme to show go-ffmpreg

* fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility

* remove extra debug from error message

* fix up incorrect function signatures

* update PutFile to just use regular file copy, as changes are file is on separate partition

* fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe

* update more tests, add more code comments

* add utilities to generate processed emoji / media outputs

* fix remaining tests

* add test for opus media file, add license header to utility cmds

* limit the number of concurrently available ffmpeg / ffprobe instances

* reduce number of instances

* further reduce number of instances

* fix envparsing test with configuration variables

* update docs and configuration with new media-{local,remote}-max-size variables
2024-07-12 09:39:47 +00:00
kim e543fbc80e
update remaining gruf libraries relying on linkname () 2024-06-21 18:36:58 +00:00
kim 9143ac6fb4
updates go-mutexes to no longer rely on unsafe linkname () 2024-06-21 17:35:32 +01:00
kim b93087ceb4
[chore] update go-structr and go-mangler to no longer rely on modern-go/reflect2 ()
* updates go-structr and go-mangler to no longer rely on modern-go/reflect2 (*phew* now we're go1.23 safe)

* update go-structr version

* bump go-structr to improve memory usage (v. slightly) in certain conditions
2024-06-21 16:43:17 +01:00
kim 6ed6824d5d
bump go-structr to v0.8.5 to improve memory usage () 2024-06-03 12:03:35 +01:00
kim 3d3e99ae52
[performance] update storage backend and make use of seek syscall when available ()
* update to use go-storage/ instead of go-store/v2/storage/

* pull in latest version from codeberg

* remove test output 😇

* add code comments

* set the exclusive bit when creating new files in disk config

* bump to actual release version

* bump to v0.1.1 (tis a simple no-logic change)

* update readme

* only use a temporary read seeker when decoding video if required (should only be S3 now)

* use fastcopy library to use memory pooled buffers when calling TempFileSeeker()

* update to use seek call in serveFileRange()
2024-05-22 11:46:24 +02:00
kim c06e6fb656
[performance] update go-structr and go-mutexes with memory usage improvements ()
* update go-structr and go-mutexes with memory usage improvements

* bump to go-structr v0.8.4
2024-05-13 08:05:46 +00:00
kim 3554991444
update go-structr -> v0.8.2 which includes some minor memory usage improvements () 2024-05-06 19:44:22 +00:00
dependabot[bot] 8237e8d09e
[chore]: Bump codeberg.org/gruf/go-structr from 0.7.0 to 0.8.0 ()
Bumps codeberg.org/gruf/go-structr from 0.7.0 to 0.8.0.

---
updated-dependencies:
- dependency-name: codeberg.org/gruf/go-structr
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-06 08:51:06 +00:00
kim c9c0773f2c
[performance] update remaining worker pools to use queues ()
* start replacing client + federator + media workers with new worker + queue types

* refactor federatingDB.Delete(), drop queued messages when deleting account / status

* move all queue purging to the processor workers

* undo toolchain updates

* code comments, ensure dereferencer worker pool gets started

* update gruf libraries in readme

* start the job scheduler separately to the worker pools

* reshuffle ordering or server.go + remove duplicate worker start / stop

* update go-list version

* fix vendoring

* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped

* add logging to worker processing functions in testrig, don't start workers in unexpected places

* update go-structr to add (+then rely on) QueueCtx{} type

* ensure more worker pools get started properly in tests

* fix remaining broken tests relying on worker queue logic

* fix account test suite queue popping logic, ensure noop workers do not pull from queue

* move back accidentally shuffled account deletion order

* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()

* silently drop deletes from accounts not permitted to

* don't warn log on forwarded deletes

* make if else clauses easier to parse

* use getFederatorMsg()

* improved code comment

* improved code comment re: requesting account delete checks

* remove boolean result from worker start / stop since false = already running or already stopped

* remove optional passed-in http.client

* remove worker starting from the admin CLI commands (we don't need to handle side-effects)

* update prune cli to start scheduler but not all of the workers

* fix rebase issues

* remove redundant return statements

* i'm sorry sir linter
2024-04-26 13:50:46 +01:00
dependabot[bot] 62788aa116
[chore]: Bump codeberg.org/gruf/go-mutexes from 1.4.0 to 1.4.1 ()
Bumps codeberg.org/gruf/go-mutexes from 1.4.0 to 1.4.1.

---
updated-dependencies:
- dependency-name: codeberg.org/gruf/go-mutexes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-22 13:43:45 +01:00
kim db2dcc3455
[chore] update go-structr => v0.6.2 (fixes nested field ptr following) ()
* update go-structr => v0.6.1 (fixes nested field ptr following)

* bump to v0.6.2
2024-04-11 11:46:08 +02:00
kim adf345f1ec
[chore] bump go structr cache version -> v0.6.0 ()
* update go-structr library -> v0.6.0, add necessary wrapping types + code changes to support these changes

* update readme with go-structr package changes

* improved wrapping of the SliceCache type

* add code comments for the cache wrapper types

* remove test.out 😇

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2024-04-02 12:03:40 +02:00
kim 81198fa2d0
update go-structr v0.2.0 => v0.3.0 to fix possible hash collision issues () 2024-01-29 15:13:53 +00:00
kim 07207e71e9
[performance] cache library performance enhancements (updates go-structr => v0.2.0) ()
* update go-structr => v0.2.0

* update readme

* whoops, fix the link
2024-01-26 12:14:10 +00:00
dependabot[bot] b3ba1516a7
[chore]: Bump codeberg.org/gruf/go-mutexes from 1.3.1 to 1.4.0 ()
Bumps codeberg.org/gruf/go-mutexes from 1.3.1 to 1.4.0.

---
updated-dependencies:
- dependency-name: codeberg.org/gruf/go-mutexes
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-22 14:33:51 +00:00
kim 7ec1e1332e
[performance] overhaul struct (+ result) caching library for simplicity, performance and multiple-result lookups ()
* rewrite cache library as codeberg.org/gruf/go-structr, implement in gotosocial

* use actual go-structr release version (not just commit hash)

* revert go toolchain changes (damn you go for auto changing this)

* fix go mod woes

* ensure %w is used in calls to errs.Appendf()

* fix error checking

* fix possible panic

* remove unnecessary start/stop functions, move to main Cache{} struct, add note regarding which caches require start/stop

* fix copy-paste artifact... 😇

* fix all comment copy-paste artifacts

* remove dropID() function, now we can just use slices.DeleteFunc()

* use util.Deduplicate() instead of collate(), move collate to util

* move orderByIDs() to util package and "generify"

* add a util.DeleteIf() function, use this to delete entries on failed population

* use slices.DeleteFunc() instead of util.DeleteIf() (i had the logic mixed up in my head somehow lol)

* add note about how collate differs from deduplicate
2024-01-19 12:57:29 +00:00
kim eb170003b8
[bugfix] return 400 Bad Request on more cases of malformed AS data () 2023-11-30 16:22:34 +00:00
dependabot[bot] d7e35f6bc9
[chore]: Bump codeberg.org/gruf/go-mutexes from 1.3.0 to 1.3.1 ()
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-27 13:20:18 +00:00
dependabot[bot] 1fa206c230
[chore]: Bump codeberg.org/gruf/go-byteutil from 1.1.2 to 1.2.0 ()
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-27 13:16:26 +00:00
kim 238e85a754
🙈 () 2023-10-31 11:30:25 +00:00
kim ce71a5a790
[feature] add per-uri dereferencer locks () 2023-10-31 11:12:22 +00:00
dependabot[bot] 70d87f0ff0
[chore]: Bump codeberg.org/gruf/go-kv from 1.6.3 to 1.6.4 () 2023-08-21 06:54:30 +00:00
kim 8ea7f551a0
[bugfix] bump go-kv version with logfmt quote fix () 2023-08-13 14:27:29 +01:00
kim 6f4ae8f58d
[bugfix] update cache library with nil ptr fix ()
Signed-off-by: kim <grufwub@gmail.com>
2023-08-06 11:55:04 +02:00
kim 9a291dea84
[performance] add caching of status fave, boost of, in reply to ID lists () 2023-08-04 12:28:33 +01:00
kim 00adf18c24
[feature] simpler cache size configuration ()
* add automatic cache max size generation based on ratios of a singular fixed memory target

Signed-off-by: kim <grufwub@gmail.com>

* remove now-unused cache max-size config variables

Signed-off-by: kim <grufwub@gmail.com>

* slight ratio tweak

Signed-off-by: kim <grufwub@gmail.com>

* remove unused visibility config var

Signed-off-by: kim <grufwub@gmail.com>

* add secret little ratio config trick

Signed-off-by: kim <grufwub@gmail.com>

* fixed a word

Signed-off-by: kim <grufwub@gmail.com>

* update cache library to remove use of TTL in result caches + slice cache

Signed-off-by: kim <grufwub@gmail.com>

* update other cache usages to use correct interface

Signed-off-by: kim <grufwub@gmail.com>

* update example config to explain the cache memory target

Signed-off-by: kim <grufwub@gmail.com>

* update env parsing test with new config values

Signed-off-by: kim <grufwub@gmail.com>

* do some ratio twiddling

Signed-off-by: kim <grufwub@gmail.com>

* add missing header

* update envparsing with latest defaults

Signed-off-by: kim <grufwub@gmail.com>

* update size calculations to take into account result cache, simple cache and extra map overheads

Signed-off-by: kim <grufwub@gmail.com>

* tweak the ratios some more

Signed-off-by: kim <grufwub@gmail.com>

* more nan rampaging

Signed-off-by: kim <grufwub@gmail.com>

* fix envparsing script

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, add sweep function to keep caches trim

Signed-off-by: kim <grufwub@gmail.com>

* sweep caches once a minute

Signed-off-by: kim <grufwub@gmail.com>

* add a regular job to sweep caches and keep under 80% utilisation

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* add new size library used to libraries section of readme

Signed-off-by: kim <grufwub@gmail.com>

* add better explanations for the mem-ratio numbers

Signed-off-by: kim <grufwub@gmail.com>

* update go-cache

Signed-off-by: kim <grufwub@gmail.com>

* library version bump

Signed-off-by: kim <grufwub@gmail.com>

* update cache.result{} size model estimation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-03 11:34:35 +02:00
kim ed2477ebea
[performance] cache follow, follow request and block ID lists () 2023-07-31 11:25:29 +01:00