mirror of
1
Fork 0
Commit Graph

417 Commits

Author SHA1 Message Date
kim c9c0773f2c
[performance] update remaining worker pools to use queues (#2865)
* start replacing client + federator + media workers with new worker + queue types

* refactor federatingDB.Delete(), drop queued messages when deleting account / status

* move all queue purging to the processor workers

* undo toolchain updates

* code comments, ensure dereferencer worker pool gets started

* update gruf libraries in readme

* start the job scheduler separately to the worker pools

* reshuffle ordering or server.go + remove duplicate worker start / stop

* update go-list version

* fix vendoring

* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped

* add logging to worker processing functions in testrig, don't start workers in unexpected places

* update go-structr to add (+then rely on) QueueCtx{} type

* ensure more worker pools get started properly in tests

* fix remaining broken tests relying on worker queue logic

* fix account test suite queue popping logic, ensure noop workers do not pull from queue

* move back accidentally shuffled account deletion order

* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()

* silently drop deletes from accounts not permitted to

* don't warn log on forwarded deletes

* make if else clauses easier to parse

* use getFederatorMsg()

* improved code comment

* improved code comment re: requesting account delete checks

* remove boolean result from worker start / stop since false = already running or already stopped

* remove optional passed-in http.client

* remove worker starting from the admin CLI commands (we don't need to handle side-effects)

* update prune cli to start scheduler but not all of the workers

* fix rebase issues

* remove redundant return statements

* i'm sorry sir linter
2024-04-26 13:50:46 +01:00
tobi b7c629a18a
[bugfix] Fix incorrect field name for status source, add helpful message (#2854)
* [bugfix] Fix incorrect field name for status source, add helpful message

* swagger

* yyammm
2024-04-18 13:22:55 +02:00
tobi 431505b3e4
[feature] Stub conversations endpoint (#2853) 2024-04-18 12:36:02 +02:00
tobi 34d0159ad5
[feature] Stub account mutes endpoint (#2852)
* [feature] Stub account mutes endpoint

* swagger? i barely know 'er!
2024-04-18 09:59:47 +01:00
tobi cef9924d9a
[feature] Status source endpoint (#2848)
* [feature] statusSource endpoint

* finish up
2024-04-17 13:49:20 +01:00
tobi ef16919d4a
[feature] Stub status history endpoint (#2847) 2024-04-17 13:06:49 +01:00
tobi 8cf685fbe9
[bugfix] Fix minor API issue w/ boosted statuses (#2846) 2024-04-17 12:41:40 +02:00
tobi 3cceed11b2
[feature/performance] Store account stats in separate table (#2831)
* [feature/performance] Store account stats in separate table, get stats from remote

* test account stats

* add some missing increment / decrement calls

* change stats function signatures

* rejig logging a bit

* use lock when updating stats
2024-04-16 13:10:13 +02:00
kim f79d50b9b2
[performance] cached oauth database types (#2838)
* update token + client code to use struct caches

* add code comments

* slight tweak to default mem ratios

* fix envparsing

* add appropriate invalidate hooks

* update the tokenstore sweeping function to rely on caches

* update to use PutClient()

* add ClientID to list of token struct indices
2024-04-15 14:22:21 +01:00
tobi 89e0cfd874
[feature] Admin accounts endpoints; approve/reject sign-ups (#2826)
* update settings panels, add pending overview + approve/deny functions

* add admin accounts get, approve, reject

* send approved/rejected emails

* use signup URL

* docs!

* email

* swagger

* web linting

* fix email tests

* wee lil fixerinos

* use new paging logic for GetAccounts() series of admin endpoints, small changes to query building

* shuffle useAccountIDIn check *before* adding to query

* fix parse from toot react error

* use `netip.Addr`

* put valid slices in globals

* optimistic updates for account state

---------

Co-authored-by: kim <grufwub@gmail.com>
2024-04-13 13:25:10 +02:00
tobi 9fb8a78f91
[feature] New user sign-up via web page (#2796)
* [feature] User sign-up form and admin notifs

* add chosen + filtered languages to migration

* remove stray comment

* chosen languages schmosen schmanguages

* proper error on local account missing
2024-04-11 11:45:53 +02:00
kim a483bd9e38
[performance] massively improved ActivityPub delivery worker efficiency (#2812)
* add delivery worker type that pulls from queue to httpclient package

* finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller

* hook up queue deletion logic

* support deleting queued http requests by target ID

* don't index APRequest by hostname in the queue

* use gorun

* use the original context's values when wrapping msg type as delivery{}

* actually log in the AP delivery worker ...

* add uncommitted changes

* use errors.AsV2()

* use errorsv2.AsV2()

* finish adding some code comments, add bad host handling to delivery workers

* slightly tweak deliveryworkerpool API, use advanced sender multiplier

* remove PopCtx() method, let others instead rely on Wait()

* shuffle things around to move delivery stuff into transport/ subpkg

* remove dead code

* formatting

* validate request before queueing for delivery

* finish adding code comments, fix up backoff code

* finish adding more code comments

* clamp minimum no. senders to 1

* add start/stop logging to delivery worker, some slight changes

* remove double logging

* use worker ptrs

* expose the embedded log fields in httpclient.Request{}

* ensure request context values are preserved when updating ctx

* add delivery worker tests

* fix linter issues

* ensure delivery worker gets inited in testrig

* fix tests to delivering messages to check worker delivery queue

* update error type to use ptr instead of value receiver

* fix test calling Workers{}.Start() instead of testrig.StartWorkers()

* update docs for advanced-sender-multiplier

* update to the latest activity library version

* add comment about not using httptest.Server{}
2024-04-11 11:45:35 +02:00
kim 83e7847cdf
fix possible nil panic (#2809) 2024-04-04 16:45:59 +01:00
kim 15ede4c1ea
[bugfix] improved authenticate post inbox error handling (#2803)
* improved PostInboxScheme() error handling / logging in case of failed auth

* dumbass kim. returning err instead of errWithCode...

* add checks for the slightly changed error handling in tests, add notes in codebase about the odd way of working
2024-04-03 13:57:07 +01:00
tobi f05874be30
[feature] Option to hide followers/following (#2788) 2024-04-02 10:42:24 +01:00
tobi 8953f57d88
[feature] User-selectable preset CSS themes for accounts (#2777)
* [feature] User-selectable preset themes

* docs, more theme stuff

* lint, tests

* fix css name

* correct some little issues

* add another theme

* fix poll background

* okay last theme i swear

* make retrieval of apimodel themes more conventional

* preallocate stylesheet slices
2024-03-25 17:32:24 +00:00
tobi b7b42e832a
[feature] Add healthcheck endpoints `/livez` and `/readyz` (#2783)
* [feature] Add healthcheck endpoints `/livez` and `/readyz`

* use select that returns no data
2024-03-25 17:05:14 +00:00
tobi 36f79e650c
[bugfix] Avoid empty public/local timeline queries (#2784) 2024-03-25 16:20:45 +00:00
tobi 7f4a0a1aeb
[chore] Move local account settings to separate db table (#2770)
* [chore] Move local account settings to separate database model

* don't use separate settings_id
2024-03-22 14:03:46 +01:00
Vyr Cossont 4a56f31047
[chore] Swagger: add missing paging params to bookmarks list (#2759)
No code changes.
2024-03-15 11:43:24 +01:00
Vyr Cossont efbc8151db
[chore] Swagger: correct names and formats for several array params (#2758)
No code changes.
2024-03-15 11:31:58 +01:00
tobi 92bf1f779b
[chore] Expose move endpoint again, small settings panel fixes (#2752) 2024-03-13 14:45:50 +01:00
tobi ab2d063fcb
[feature] Process outgoing Move from clientAPI (#2750)
* prevent moved accounts from taking create-type actions

* update move logic

* federate move out

* indicate on web profile when an account has moved

* [docs] Add migration docs section

* lock while checking + setting move state

* use redirectFollowers func for clientAPI as well

* comment typo

* linter? i barely know 'er!

* Update internal/uris/uri.go

Co-authored-by: Daenney <daenney@users.noreply.github.com>

* add a couple tests for move

* fix little mistake exposed by tests (thanks tests)

* ensure Move marked as successful

* attach shared util funcs to struct

* lock whole account when doing move

* move moving check to after error check

* replace repeated text with error func

* linterrrrrr!!!!

* catch self follow case

---------

Co-authored-by: Daenney <daenney@users.noreply.github.com>
2024-03-13 13:53:29 +01:00
Vyr Cossont 13b9fd5f92
[bugfix]: Add missing Link headers in Swagger spec (#2751)
* Adds Link headers to all endpoints that use it for paging
* Fixes the return type for listing what accounts someone follows
2024-03-13 12:23:26 +01:00
Vyr Cossont 74e84cf8fa
[docs/chore] Swagger fixes for filters (#2730)
* Swagger: filtersV1Get should return an array

* Swagger: context must use multi format
2024-03-07 11:28:57 +01:00
Vyr Cossont 5159664a51
[chore] Fix a Swagger warning that only manifests during Go client code generation (#2729) 2024-03-06 18:44:58 +01:00
Vyr Cossont fc3741365c
[bugfix] Fix Swagger spec and add test script (#2698)
* Add Swagger spec test script

* Fix Swagger spec errors not related to statuses with polls

* Add API tests that post a status with a poll

* Fix creating a status with a poll from form params

* Fix Swagger spec errors related to statuses with polls (this is the last error)

* Fix Swagger spec warnings not related to unused definitions

* Suppress a duplicate list update params definition that was somehow causing wrong param names

* Add Swagger test to CI

- updates Drone config
- vendorizes go-swagger
- fixes a file extension issue that caused the test script to generate JSON instead of YAML with the vendorized version

* Put `Sample: ` on its own line everywhere

* Remove unused id param from emojiCategoriesGet

* Add 5 more pairs of profile fields to account update API Swagger

* Remove Swagger prefix from dummy fields

It makes the generated code look weird

* Manually annotate params for statusCreate operation

* Fix all remaining Swagger spec warnings

- Change some models into operation parameters
- Ignore models that already correspond to manually documented operation parameters but can't be trivially changed (those with file fields)

* Documented that creating a status with scheduled_at isn't implemented yet

* sign drone.yml

* Fix filter API Swagger errors

* fixup! Fix filter API Swagger errors

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-03-06 18:05:45 +01:00
Vyr Cossont 61a2b91f45
[feature] Filters v1 (#2594)
* Implement client-side v1 filters

* Exclude linter false positives

* Update test/envparsing.sh

* Fix minor Swagger, style, and Bun usage issues

* Regenerate Swagger

* De-generify filter keywords

* Remove updating filter statuses

This is an operation that the Mastodon v2 filter API doesn't actually have, because filter statuses, unlike keywords, don't have options: the only info they contain is the status ID to be filtered.

* Add a test for filter statuses specifically

* De-generify filter statuses

* Inline FilterEntry

* Use vertical style for Bun operations consistently

* Add comment on Filter DB interface

* Remove GoLand linter control comments

Our existing linters should catch these, or they don't matter very much

* Reduce memory ratio for filters
2024-03-06 11:15:58 +01:00
kim fcecd0c952
[bugfix] unwrap boosts when checking in-reply-to status (#2702)
* add stronger checks on status being replied to

* update error code test is expecting
2024-02-29 15:20:57 +01:00
9p4 9bf448be7a
[feature/oidc] Add support for very basic RBAC (#2642)
* Add support for very basic RBAC

* Add some small tests for allowedGroup and adminGroup

* Switch to table-driven tests
2024-02-27 16:07:29 +01:00
tobi 9cadc764b3
[feature] Add experimental `instance-federation-spam-filter` option (#2685)
* [chore] Move `visibility` to `filter/visibility`

* [feature] Add experimental instance-federation-spam-filter option
2024-02-27 12:22:05 +00:00
tobi 4b0eefbcc9
[chore] Increase default max image description to 1500 chars, collapse cw char count into status (#2682)
* [chore] Make default max image description 1500 chars, collapse cw char count into status

* oops

* tests
2024-02-23 18:28:09 +00:00
kim 291e180990
[bugfix] fix possible mutex lockup during streaming code (#2633)
* rewrite Stream{} to use much less mutex locking, update related code

* use new context for the stream context

* ensure stream gets closed on return of writeTo / readFrom WSConn()

* ensure stream write timeout gets cancelled

* remove embedded context type from Stream{}, reformat log messages for consistency

* use c.Request.Context() for context passed into Stream().Open()

* only return 1 boolean, fix tests to expect multiple stream types in messages

* changes to ping logic

* further improved ping logic

* don't export unused function types, update message sending to only include relevant stream type

* ensure stream gets closed 🤦

* update to error log on failed json marshal (instead of panic)

* inverse websocket read error checking to _ignore_ expected close errors
2024-02-20 18:07:49 +00:00
tobi 8cafa6b74b
[feature] Add `requested_by` to relationship model (#2672)
* [feature] Add `requested_by` to relationship model

* whoops, missed some tests
2024-02-20 17:50:54 +00:00
tobi d10226e912
[bugfix] Use ptr for instance stats entries to avoid skipping 0 values (#2666)
* [bugfix] Use ptr for instance stats entries to avoid skipping 0 values

* comment explaining why stats values are pointers
2024-02-19 13:17:14 +01:00
tobi 83a4adbd95
[chore] Disable Move API endpoints for now until Move is fully implemented in the backend (#2650)
* [chore] Disable Move API endpoints for now until Move is fully implemented in the backend

* disable other form fields

* clarify that moving to GtS also isn't implemented yet
2024-02-17 17:03:44 +01:00
kim 8b8211986e
[chore] also allow text/xml in place of application/xml (#2640) 2024-02-14 13:07:58 +01:00
kim 2bafd7daf5
[bugfix] add stricter checks during all stages of dereferencing remote AS objects (#2639)
* add stricter checks during all stages of dereferencing remote AS objects

* a comment
2024-02-14 12:13:38 +01:00
tobi aa8bbe6ad2
[bugfix] Fix Postgres emoji delete, emoji category change (#2570)
* [bugfix] Fix Postgres emoji delete, emoji category change

* revert trace logging

* caching issue

* update tests
2024-01-29 14:57:22 +00:00
tobi 14b684b2b5
[chore] Add a couple tests for updating list entries (#2580) 2024-01-28 13:01:12 +01:00
Vyr Cossont 7e0a203173
[bugfix] Fix EmptyJSONObject/EmptyJSONArray (#2576)
* Fix EmptyJSONObject/EmptyJSONArray

These are meant to be the bytes representing an empty object and array in JSON: `{}` and `[]`. They are actually the strings `"{}"` and `"[]"`. This causes clients expecting an object or array to not be able to parse the response.

* Use json.RawMessage instead of []byte
2024-01-28 11:49:04 +01:00
tobi 138cbe4d60
[feature] Ratelimit + serve emoji images on separate router group (#2548)
* [feature] Serve + rate limit emoji files separately from attachments

* add a wee little warning about uploading loads of emojis
2024-01-22 15:17:04 +00:00
kim 67e11a1a61
[chore] chore rationalise http return codes for activitypub handlers (#2540)
* some small code fixups and changes

* add check in ResolveIncomingActivity for transient activity types (i.e. activity ID is nil)

* update test to handle new transient behaviour
2024-01-18 17:11:13 +01:00
kim 906639ad7e
[chore] update viper version (#2539)
* update viper version

* removes our last uses of the slice package

* fix tests
2024-01-17 14:54:30 +00:00
tobi c36f9ac37b
[feature] Account alias / move API + db models (#2518)
* [feature] Account alias / move API + db models

* go fmt

* fix little cherry-pick issues

* update error checking, formatting

* add and use new util functions to simplify alias logic
2024-01-16 16:22:44 +00:00
tobi d5e3996a18
[feature] Parse instance descriptors as markdown, show T&C on /about (#2481)
* [feature] Parse instance descriptors as markdown, show T&C on /about

* lint

* remove unnecessary nullzero tags
2024-01-05 13:39:31 +01:00
tobi 0ff52b71f2
[chore] Refactor HTML templates and CSS (#2480)
* [chore] Refactor HTML templates and CSS

* eslint

* ignore "Local"

* rss tests

* fiddle with OG just a tiny bit

* dick around with polls a bit more so SR stops saying "clickable"

* remove break

* oh lord

* don't lazy load avatar

* fix ogmeta tests

* clean up some cruft

* catch remaining calls to c.HTML

* fix error rendering + stack overflow in tag

* allow templating attributes

* fix indent

* set aria-hidden on status complementary content, since it's already present in the label anyway

* tidy up templating calls a little

* try to make styling a bit more consistent + readable

* fix up some remaining CSS issues

* fix up reports
2023-12-27 11:23:52 +01:00
kim 97a1fd9a29
[bugfix] 😇 (#2476) 2023-12-18 16:40:29 +00:00
kim 8ebb7775a3
[feature] request blocking by http headers (#2409) 2023-12-18 14:18:25 +00:00
kim d56a8d095e
[performance] simpler throttling logic (#2407)
* reduce complexity of throttling logic to use 1 queue and an atomic int

* use atomic add instead of CAS, add throttling test
2023-12-16 12:53:42 +01:00