mirror of
1
Fork 0
Commit Graph

361 Commits

Author SHA1 Message Date
tobi 5543fd5340
[feature/frontend] Add options to include Unlisted posts or hide all posts ()
* [feature/frontend] Add options to include Unlisted posts or hide all posts

* finish up

* swagger

* move invalidate call into bundb package, avoid invalidating if not necessary

* rename show_web_statuses => web_visibility

* don't use ptr for webvisibility

* last bits
2024-09-09 18:07:25 +02:00
Vyr Cossont 100d660797
[bugfix] Swagger: add missing filter_action param for v2 filter PUT () 2024-09-09 09:52:49 +02:00
kim 0a1555521d
[performance] use single-threaded image transforms ()
* use single-threaded image resizing in native code so we have more control over goroutines

* implement parallel-free versions of image transform functions also

* remove debug code
2024-08-31 10:41:38 +02:00
tobi e10aa76612
[chore] Only call `imaging.Resize when necessary`, use even tinier blurhashes ()
* [chore] Use `imaging.Fit`, use even tinier blurhashes

* avoid calling resize if not necessary

* update blurhashes + thumb
2024-08-29 17:43:14 +02:00
tobi f23f04e0b1
[feature] Interaction requests client api + settings panel ()
* [feature] Interaction requests client api + settings panel

* test accept / reject

* fmt

* don't pin rejected interaction

* use single db model for interaction accept, reject, and request

* swaggor

* env sharting

* append errors

* remove ErrNoEntries checks

* change intReqID to reqID

* rename "pend" to "request"

* markIntsPending -> mark interactionsPending

* use log instead of returning error when rejecting interaction

* empty migration

* jolly renaming

* make interactionURI unique again

* swag grr

* remove unnecessary locks

* invalidate as last step
2024-08-24 11:49:37 +02:00
tobi 53fccb8af8
[feature] Use `local_only` field, deprecate `federated` field ()
* [feature] Use `local_only` field, deprecate `federated` field

* use `deprecated` comment for form.Federated

* nolint
2024-08-22 19:47:10 +02:00
kim f77005128a
[performance] move thumbnail generation to go code where possible ()
* wrap thumbnailing code to handle generation natively where possible

* more code comments!

* add even more code comments!

* add code comments about blurhash generation

* maintain image rotation if contained in exif data

* move rotation before resizing

* ensure pix_fmt actually selected by ffprobe, check for alpha layer with gifs

* use linear instead of nearest-neighbour for resizing

* work with image "orientation" instead of "rotation". use default 75% quality for both webp and jpeg generation

* add header to new file

* use thumb extension when getting thumb mime type

* update test models and tests with new media processing

* add suggested code comments

* add note about thumbnail filter count reducing memory usage
2024-08-08 17:12:13 +00:00
Leah Neukirchen 4697271cef
[bugfix] send back Sec-Websocket-Protocol header for streaming WebSocket ()
* [bugfix] send back Sec-Websocket-Protocol header for streaming WebSocket

Chrome expects the selected Sec-Websocket-Protocol to be sent back
on the WebSocket upgrade request (RFC6455 1.9).

* fiddle a bit to avoid getting headers multiple times

* add some explanatory notes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-06 11:52:27 +02:00
kim b85a9983d0
[bugfix] fix emoji recaching operations ()
* add test for emoji update image

* update emoji recache to set the instance account id

* don't refresh emoji if only not cached. in that case literally just recache

* code comment

* rename + move a few things

* add some more code comments, and rename some functions to make logic a bit clearer

* remove unnecessary nil check (the value can be nil)

* comment wording

* remove test data output

* handle the case of caching an emoji which has been refreshed then uncached

* allow overwriting on testrig storage as we do now on regular storage

* fix emoji category ID not getting updated

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-03 19:05:38 +02:00
tobi 7b5917d6ae
[feature] Allow import of following and blocks via CSV ()
* [feature] Import follows + blocks via settings panel

* test import follows
2024-08-02 12:41:46 +01:00
Vyr Cossont fd837776e2
[feature] Implement Mastodon-compatible roles ()
* Implement Mastodon-compatible roles

- `Account.role` should only be available through verify_credentials for checking current user's permissions
- `Account.role` now carries a Mastodon-compatible permissions bitmap and a marker for whether it should be shown to the public
- `Account.roles` added for *public* display roles (undocumented but stable since Mastodon 4.1)
- Web template now uses only public display roles (no user-visible change here, we already special-cased the `user` role)

* Handle verify_credentials case for default role

* Update JSON exact-match tests

* Address review comments

* Add blocks bit to admin permissions bitmap
2024-07-31 09:26:09 -07:00
tobi 38f041cea1
[feature] Allow users to export data via the settings panel ()
* [feature] Allow users to export data via the settings panel

* rename/move some stuff
2024-07-31 15:03:34 +01:00
Vyr Cossont a237e2b295
[feature] Implement following hashtags ()
* Implement followed tags API

* Insert statuses with followed tags into home timelines

* Test following and unfollowing tags

* Correct Swagger path params

* Trim conversation caches

* Migration for followed_tags table

* Followed tag caches and DB implementation

* Lint and tests

* Add missing tag info endpoint, reorganize tag API

* Unwrap boosts when timelining based on tags

* Apply visibility filters to tag followers

* Address review comments
2024-07-29 19:26:31 +01:00
tobi c9b6220fef
[chore] Add interaction filter to complement existing visibility filter ()
* [chore] Add interaction filter to complement existing visibility filter

* pass in ptr to visibility and interaction filters to Processor{} to ensure shared

* use int constants for for match type, cache db calls in filterctx

* function name typo 😇

---------

Co-authored-by: kim <grufwub@gmail.com>
2024-07-24 12:27:42 +01:00
Vyr Cossont 86a59db711
Remove content and related fields from boosts ()
These duplicate the content of the target and aren't necessary for anything.

- Stops copying some fields from target when boosting or processing a remote boost
- Adds a migration to null out existing duplicate data
- Updates tests
2024-07-23 20:51:57 +01:00
Vyr Cossont 8fdd358f4b
[feature] Conversations API ()
* Implement conversations API

* Sort and page conversations by last status ID

* Appease linter

* Fix deleting conversations and statuses

* Refactor to make migrations automatic

* Lint

* Update tests post-merge

* Fixes from live-fire testing

* Linter caught a format problem

* Refactor tests, fix cache

* Negative test for non-DMs

* Run conversations advanced migration on testrig startup as well as regular server startup

* Document (lack of) side effects of API method for deleting a conversation

* Make not-found check less nested for readability

* Rename PutConversation to UpsertConversation

* Use util.Ptr instead of IIFE

* Reduce cache used by conversations

* Remove unnecessary TableExpr/ColumnExpr

* Use struct tags for both unique constraints on Conversation

* Make it clear how paging with GetDirectStatusIDsBatch should be used

* Let conversation paging skip conversations it can't render

* Use Bun NewDropTable

* Convert delete raw query to Bun

* Convert update raw query to Bun

* Convert latestConversationStatusesTempTable raw query partially to Bun

* Convert conversationStatusesTempTable raw query partially to Bun

* Rename field used to store result of MaxDirectStatusID

* Move advanced migrations to their own tiny processor

* Catch up util function name with main

* Remove json.… wrappers

* Remove redundant check

* Combine error checks

* Replace map with slice of structs

* Address processor/type converter comments

- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation

* Add error context when dropping temp tables
2024-07-23 20:44:31 +01:00
kim c635080a0b
[feature] add flac support ()
* add flac support to the ffprobe format/stream -> filetype parser

* also add audio/flac for flac (not just x-flac)

* update tests

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-21 14:42:51 +02:00
tobi 027a93facc
[feature/frontend] Respect `prefers-reduced-motion` for avatars, headers, and emojis ()
* [feature/frontend] Respect `prefers-reduced-motion` for avatars, headers, and emojis

* go fmt

* fix tests

* use static version of instance thumbnail when appropriate

* use prefers-reduced-motion

* simplify account conversion a bit

* fix c&p error
2024-07-21 14:22:08 +02:00
tobi ad700c5029
[chore/frontend] Use smaller webp images for logo and default avatars () 2024-07-20 14:02:22 +01:00
kim 50c9b5498b
[feature] use webp for thumbnails ()
* update to use webp for thumbnails

* bump webp quality up to 40% from 12% (it's a bit different to jpeg quality setting)

* update to use yuva colorspace, and use thumbnail=n=10 to select frame

* fix missing comma in ffmpeg args

* add links to appropriate ffmpeg docs

* update tests

* add file size tests for thumbnails

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-19 17:28:43 +02:00
kim 72ba5666a6
[chore] media pipeline improvements ()
* don't set emoji / media image paths on failed download, migrate FileType from string to integer

* fix incorrect uses of util.PtrOr, fix returned frontend media

* fix migration not setting arguments correctly in where clause

* fix not providing default with not null column

* whoops

* ensure a default gets set for media attachment file type

* remove the exclusive flag from writing files in disk storage

* rename PtrOr -> PtrOrZero, and rename PtrValueOr -> PtrOrValue to match

* slight wording changes

* use singular / plural word forms (no parentheses), is better for screen readers

* update testmodels with unknown media type to have unset file details, update attachment focus handling converting to frontend, update tests

* store first instance in ffmpeg wasm pool, fill remaining with closed instances
2024-07-17 15:26:33 +00:00
tobi 0aadc2db2a
[feature] Allow users to set default interaction policies per status visibility ()
* [feature] Allow users to set default interaction policies

* use vars for default policies

* avoid some code repetition

* unfuck form binding

* avoid bonkers loop

* beep boop

* put policyValsToAPIPolicyVals in separate function

* don't bother with slices.Grow

* oops
2024-07-17 16:46:52 +02:00
kim de45c0be60
[feature] more filetype support! ()
* add more supported file types to our media processor that ffmpeg supports, update supported mime type lists

* add code comments to the supported mime types slice

* don't check for zero value string, just parse

* remove some unneeded consts which make the code a bit harder to read

* fix test expected instance media mime types, use compact ffprobe json, simple media processing by type

* final tweaks to media processing code

* don't use safe divide where we don't need to
2024-07-15 15:24:53 +01:00
tobi 9efb11d848
[feature/frontend] Add player for audio files; use thumbnail for `poster` ()
* [feature/frontend] Audio player for audio media types

* use video preview images for previews instead of video itself

* don't preload

* update tests for new zork status

* collapse media gallery into single row when small
2024-07-15 10:47:57 +01:00
tobi aeb65bceae
[feature/frontend] Better visual separation between "main" thread and "replies" ()
* [feature/frontend] Better web threading model

* fix test

* bwap

* tweaks

* more tweaks to wording

* typo

* indenting

* adjust wording

* aaa
2024-07-12 20:36:03 +02:00
kim cde2fb6244
[feature] support processing of (many) more media types ()
* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg

* specify the video codec to use when generating static image from emoji

* update go-storage library (fixes incompatibility after updating go-iotools)

* maintain image aspect ratio when generating a thumbnail for it

* update readme to show go-ffmpreg

* fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility

* remove extra debug from error message

* fix up incorrect function signatures

* update PutFile to just use regular file copy, as changes are file is on separate partition

* fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe

* update more tests, add more code comments

* add utilities to generate processed emoji / media outputs

* fix remaining tests

* add test for opus media file, add license header to utility cmds

* limit the number of concurrently available ffmpeg / ffprobe instances

* reduce number of instances

* further reduce number of instances

* fix envparsing test with configuration variables

* update docs and configuration with new media-{local,remote}-max-size variables
2024-07-12 09:39:47 +00:00
tobi 5bc567196b
[chore] Add interaction policy gtsmodels ()
* [chore] introduce interaction policy gts models

* update migration a smidge

* fix copy paste typo

* update migration

* use int for InteractionType
2024-07-11 15:44:29 +01:00
tobi d70f4e166d
[feature/frontend] Allow setting alt-text for avatar + header () 2024-07-08 15:47:03 +02:00
kim 21bb324156
[chore] media and emoji refactoring ()
* start updating media manager interface ready for storing attachments / emoji right away

* store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load()

* remove now unused media workers

* fix tests and issues

* fix another test!

* fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues

* fix more tests

* fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis

* whoops, rebase issue

* remove kim's whacky experiments

* do some reshuffling, ensure emoji uri gets set

* ensure marked as not cached on cleanup

* tweaks to media / emoji processing to handle context canceled better

* ensure newly fetched emojis actually get set in returned slice

* use different varnames to be a bit more obvious

* move emoji refresh rate limiting to dereferencer

* add exported dereferencer functions for remote media, use these for recaching in processor

* add check for nil attachment in updateAttachment()

* remove unused emoji and media fields + columns

* see previous commit

* fix old migrations expecting image_updated_at to exists (from copies of old models)

* remove freshness checking code (seems to be broken...)

* fix error arg causing nil ptr exception

* finish documentating functions with comments, slight tweaks to media / emoji deref error logic

* remove some extra unneeded boolean checking

* finish writing documentation (code comments) for exported media manager methods

* undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot

* move doesColumnExist() to util.go in migrations package
2024-06-26 16:01:16 +01:00
tobi 4ce5c37df5
[bugfix] rename `include_types[]` to `types[]` () 2024-06-18 18:18:35 +02:00
tobi d2b3d37724
[feature/frontend] Reports frontend v2 ()
* use apiutil + paging in admin processor+handlers

* we're making it happen

* fix little whoopsie

* styling for report list

* don't youuuu forget about meee don't don't don't don't

* last bits

* sanitize content before showing in report statuses

* update report docs
2024-06-18 18:18:00 +02:00
Vyr Cossont b08c1bd0cb
[feature] Implement types[] param for notifications ()
Counterpart of exclude_types[].

Also updates Swagger spec for types[] to use the correct param name and enumerate possible values.

Fixes 
2024-06-17 20:50:50 +01:00
Vyr Cossont b789fe2bc7
[feature] filter API v2: Restore keywords_attributes and statuses_attributes ()
These filter API v2 features were cut late in development because the form encoding version is hard to implement correctly and because I thought no clients actually used `keywords_attributes`. Unfortunately, Phanpy does use `keywords_attributes`.
2024-06-14 10:11:41 +02:00
Vyr Cossont dcf825bd87
[bugfix] Correct Swagger path for poll voting ()
Updates the Swagger spec to match the router.

Fixes 
2024-06-12 11:16:46 +02:00
tobi ce3b8aacf7
[chore] Warn about email/password change when using OIDC ()
* [chore] Warn about email/password change when using OIDC

* go fmt
2024-06-07 15:21:57 +01:00
Vyr Cossont afcfa48a7d
[feature] Implement filters_changed stream event () 2024-06-07 10:51:13 +02:00
Vyr Cossont e605788b4b
[bugfix] Filter fixes ()
* Add Filter.Expired convenience method, consistent with mutes

* Add missing Swagger for filter keyword/status delete and for 403s on moved accounts
2024-06-06 20:16:20 +02:00
Vyr Cossont 5e2d4fdb19
[feature] User muting ()
* User muting

* Address review feedback

* Rename uniqueness constraint on user_mutes to match convention

* Remove unused account_id from where clause

* Add UserMute to NewTestDB

* Update test/envparsing.sh with new and fixed cache stuff

* Address tobi's review comments

* Make compiledUserMuteListEntry.expired consistent with UserMute.Expired

* Make sure mute_expires_at is serialized as an explicit null for indefinite mutes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-06 16:38:02 +00:00
tobi bcda048eab
[feature] Self-serve email change for users ()
* [feature] Email change

* frontend stuff for changing email

* docs

* tests etc

* differentiate more clearly between local user+account and account

* populate user
2024-06-06 14:43:25 +01:00
Vyr Cossont 04bcde08a1
[feature] Add from: search operator and account_id query param ()
* Add from: search operator

* Fix whitespace in Swagger YAML comment

* Move query parsing into its own method

* Document search

* Clarify post search scope
2024-05-31 12:57:42 +02:00
Vyr Cossont 61a8d36255
[feature] Implement Filter API v2 ()
* Use correct entity name

* We support server-side filters now

* Document filter v1 methods that can throw a 409

* Validate v1 filter phrase as filter title

* Always check v1 filter API status codes in tests

* Document keyword minimum requirement on filter API v1

* Make it possible to specify filter keyword update columns per filter keyword

* Implement v2 filter API

* Fix lint and tests

* Update Swagger spec

* Fix filter update test

* Update Swagger spec *correctly*

* Update actual files Swagger spec was generated from

* Remove keywords_attributes and statuses_attributes

* Add test for serialization of empty filter

* More helpful messages when object is owned by wrong account
2024-05-31 12:55:56 +02:00
Vyr Cossont 975e92b7f1
[feature] Implement profile API ()
* Implement profile API

This Mastodon 4.2 extension provides capabilities missing from the existing Mastodon account update API: deleting an account's avatar or header.

See: https://docs.joinmastodon.org/methods/profile/

* Move profile media methods to media processor

* Remove check for moved account
2024-05-29 12:57:44 +02:00
tobi f9a4a6120d
[feature] Debug admin endpoint to clear caches ()
* [feature] Debug admin endpoint to clear caches

* go fmt
2024-05-29 12:56:17 +02:00
tobi a276b1ca06
[feature/frontend] Let admins send test email to validate SMTP config ()
* [feature/frontend] Let admins send test email to validate SMTP config

* wee
2024-05-27 17:03:54 +00:00
kim 1e7b32490d
[experiment] add alternative wasm sqlite3 implementation available via build-tag ()
This allows for building GoToSocial with [SQLite transpiled to WASM](https://github.com/ncruces/go-sqlite3) and accessed through [Wazero](https://wazero.io/).
2024-05-27 17:46:15 +02:00
kim 3d3e99ae52
[performance] update storage backend and make use of seek syscall when available ()
* update to use go-storage/ instead of go-store/v2/storage/

* pull in latest version from codeberg

* remove test output 😇

* add code comments

* set the exclusive bit when creating new files in disk config

* bump to actual release version

* bump to v0.1.1 (tis a simple no-logic change)

* update readme

* only use a temporary read seeker when decoding video if required (should only be S3 now)

* use fastcopy library to use memory pooled buffers when calling TempFileSeeker()

* update to use seek call in serveFileRange()
2024-05-22 11:46:24 +02:00
tobi 6171dcbe51
[feature] Add HTTP header permission section to frontend ()
* [feature] Add HTTP header filter section to frontend

* tweak naming a bit
2024-05-05 11:47:22 +00:00
tobi 725a21b027
[feature] Page through accounts as moderator ()
* [feature] Page through accounts as moderator

* aaaaa

* use COLLATE "C" for Postgres to ensure same ordering as SQLite

* fix typo, test paging up

* don't show moderation / info for our instance acct
2024-05-01 14:11:22 +01:00
kim a8254a40e7
[bugfix] further paging mishaps ()
* FURTHER paging shenanigans 🥲

* remove cursor logic from ToLinkURL()

* fix up paging tests

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-04-30 16:22:23 +02:00
kim c9c0773f2c
[performance] update remaining worker pools to use queues ()
* start replacing client + federator + media workers with new worker + queue types

* refactor federatingDB.Delete(), drop queued messages when deleting account / status

* move all queue purging to the processor workers

* undo toolchain updates

* code comments, ensure dereferencer worker pool gets started

* update gruf libraries in readme

* start the job scheduler separately to the worker pools

* reshuffle ordering or server.go + remove duplicate worker start / stop

* update go-list version

* fix vendoring

* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped

* add logging to worker processing functions in testrig, don't start workers in unexpected places

* update go-structr to add (+then rely on) QueueCtx{} type

* ensure more worker pools get started properly in tests

* fix remaining broken tests relying on worker queue logic

* fix account test suite queue popping logic, ensure noop workers do not pull from queue

* move back accidentally shuffled account deletion order

* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()

* silently drop deletes from accounts not permitted to

* don't warn log on forwarded deletes

* make if else clauses easier to parse

* use getFederatorMsg()

* improved code comment

* improved code comment re: requesting account delete checks

* remove boolean result from worker start / stop since false = already running or already stopped

* remove optional passed-in http.client

* remove worker starting from the admin CLI commands (we don't need to handle side-effects)

* update prune cli to start scheduler but not all of the workers

* fix rebase issues

* remove redundant return statements

* i'm sorry sir linter
2024-04-26 13:50:46 +01:00