* [chore] Synchronise our robots.txt with upstream
* [feature] Add headers to escape AI crawlers
This adds 2 headers that a number of AI crawlers respect to signal that
content should not be included in their datasets.
* add back exif-terminator and use only for jpeg,png,webp
* fix arguments passed to terminateExif()
* pull in latest exif-terminator
* fix test
* update processed img
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* Implement Mastodon-compatible roles
- `Account.role` should only be available through verify_credentials for checking current user's permissions
- `Account.role` now carries a Mastodon-compatible permissions bitmap and a marker for whether it should be shown to the public
- `Account.roles` added for *public* display roles (undocumented but stable since Mastodon 4.1)
- Web template now uses only public display roles (no user-visible change here, we already special-cased the `user` role)
* Handle verify_credentials case for default role
* Update JSON exact-match tests
* Address review comments
* Add blocks bit to admin permissions bitmap
* persist queued worker tasks to database on shutdown, fill worker queues from database on startup
* ensure the tasks are sorted by creation time before pushing them
* add migration to insert WorkerTask{} into database, add test for worker task persistence
* add test for recovering worker queues from database
* quick tweak
* whoops we ended up with double cleaner job scheduling
* insert each task separately, because bun is throwing some reflection error??
* add specific checking of cancelled worker contexts
* add http request signing to deliveries recovered from database
* add test for outgoing public key ID being correctly set on delivery
* replace select with Queue.PopCtx()
* get rid of loop now we don't use it
* remove field now we don't use it
* ensure that signing func is set
* header values weren't being copied over 🤦
* use ptr for httpclient.Request in delivery
* move worker queue filling to later in server init process
* fix rebase issues
* make logging less shouty
* use slices.Delete() instead of copying / reslicing
* have database return tasks in ascending order instead of sorting them
* add a 1 minute timeout to persisting worker queues
* Implement followed tags API
* Insert statuses with followed tags into home timelines
* Test following and unfollowing tags
* Correct Swagger path params
* Trim conversation caches
* Migration for followed_tags table
* Followed tag caches and DB implementation
* Lint and tests
* Add missing tag info endpoint, reorganize tag API
* Unwrap boosts when timelining based on tags
* Apply visibility filters to tag followers
* Address review comments
* take into account rotation when generating thumbnail, simplify ffprobe output to show only fields we need
* only show rotation side data
* remove unnecessary comment
* fix code comments
* remove debug logging
* [feature] Federate interaction policies + Accepts; enforce policies
* use Acceptable type
* fix index
* remove appendIRIStrs
* add GetAccept federatingdb function
* lock on object IRI
* [chore] Add interaction filter to complement existing visibility filter
* pass in ptr to visibility and interaction filters to Processor{} to ensure shared
* use int constants for for match type, cache db calls in filterctx
* function name typo 😇
---------
Co-authored-by: kim <grufwub@gmail.com>
These duplicate the content of the target and aren't necessary for anything.
- Stops copying some fields from target when boosting or processing a remote boost
- Adds a migration to null out existing duplicate data
- Updates tests
* Implement conversations API
* Sort and page conversations by last status ID
* Appease linter
* Fix deleting conversations and statuses
* Refactor to make migrations automatic
* Lint
* Update tests post-merge
* Fixes from live-fire testing
* Linter caught a format problem
* Refactor tests, fix cache
* Negative test for non-DMs
* Run conversations advanced migration on testrig startup as well as regular server startup
* Document (lack of) side effects of API method for deleting a conversation
* Make not-found check less nested for readability
* Rename PutConversation to UpsertConversation
* Use util.Ptr instead of IIFE
* Reduce cache used by conversations
* Remove unnecessary TableExpr/ColumnExpr
* Use struct tags for both unique constraints on Conversation
* Make it clear how paging with GetDirectStatusIDsBatch should be used
* Let conversation paging skip conversations it can't render
* Use Bun NewDropTable
* Convert delete raw query to Bun
* Convert update raw query to Bun
* Convert latestConversationStatusesTempTable raw query partially to Bun
* Convert conversationStatusesTempTable raw query partially to Bun
* Rename field used to store result of MaxDirectStatusID
* Move advanced migrations to their own tiny processor
* Catch up util function name with main
* Remove json.… wrappers
* Remove redundant check
* Combine error checks
* Replace map with slice of structs
* Address processor/type converter comments
- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation
* Add error context when dropping temp tables
* refactor file handling a tiny bit
* whoops
* make processing media / emoji defers a bit clear to see that it's the "on finished processing" path
* some wording
* add some debug logging
* add mutex locks for processing remote media
* try removing freshness check
* fix derefMedia not being allocated
* fix log format string
* handle case of empty file paths (i.e. not stored)
* remove media / emoji once finished processing from dereferencer maps
* whoops, fix the cached / force checks
* move url parsing outside of 'process___Safely()' funcs to prevalidate url
* use emoji.ShortcodeDomain()
* update RefreshEmoji() to also match RefreshMedia() changes
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* add flac support to the ffprobe format/stream -> filetype parser
* also add audio/flac for flac (not just x-flac)
* update tests
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* [feature/frontend] Respect `prefers-reduced-motion` for avatars, headers, and emojis
* go fmt
* fix tests
* use static version of instance thumbnail when appropriate
* use prefers-reduced-motion
* simplify account conversion a bit
* fix c&p error
* update to use webp for thumbnails
* bump webp quality up to 40% from 12% (it's a bit different to jpeg quality setting)
* update to use yuva colorspace, and use thumbnail=n=10 to select frame
* fix missing comma in ffmpeg args
* add links to appropriate ffmpeg docs
* update tests
* add file size tests for thumbnails
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* don't set emoji / media image paths on failed download, migrate FileType from string to integer
* fix incorrect uses of util.PtrOr, fix returned frontend media
* fix migration not setting arguments correctly in where clause
* fix not providing default with not null column
* whoops
* ensure a default gets set for media attachment file type
* remove the exclusive flag from writing files in disk storage
* rename PtrOr -> PtrOrZero, and rename PtrValueOr -> PtrOrValue to match
* slight wording changes
* use singular / plural word forms (no parentheses), is better for screen readers
* update testmodels with unknown media type to have unset file details, update attachment focus handling converting to frontend, update tests
* store first instance in ffmpeg wasm pool, fill remaining with closed instances
* [feature] Allow users to set default interaction policies
* use vars for default policies
* avoid some code repetition
* unfuck form binding
* avoid bonkers loop
* beep boop
* put policyValsToAPIPolicyVals in separate function
* don't bother with slices.Grow
* oops
* add more supported file types to our media processor that ffmpeg supports, update supported mime type lists
* add code comments to the supported mime types slice
* don't check for zero value string, just parse
* remove some unneeded consts which make the code a bit harder to read
* fix test expected instance media mime types, use compact ffprobe json, simple media processing by type
* final tweaks to media processing code
* don't use safe divide where we don't need to
* [feature/frontend] Audio player for audio media types
* use video preview images for previews instead of video itself
* don't preload
* update tests for new zork status
* collapse media gallery into single row when small
* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg
* specify the video codec to use when generating static image from emoji
* update go-storage library (fixes incompatibility after updating go-iotools)
* maintain image aspect ratio when generating a thumbnail for it
* update readme to show go-ffmpreg
* fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility
* remove extra debug from error message
* fix up incorrect function signatures
* update PutFile to just use regular file copy, as changes are file is on separate partition
* fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe
* update more tests, add more code comments
* add utilities to generate processed emoji / media outputs
* fix remaining tests
* add test for opus media file, add license header to utility cmds
* limit the number of concurrently available ffmpeg / ffprobe instances
* reduce number of instances
* further reduce number of instances
* fix envparsing test with configuration variables
* update docs and configuration with new media-{local,remote}-max-size variables
* Set frame-ancestors in the CSP
This ensures we can't be loaded/embedded in an iframe. It also sets the
older X-Frame-Options for fallback.
* Disable MIME type sniffing
* Set Referrer-Policy
This sets the policy such that browsers will never send the Referer
header along with a request, unless it's a request to the same protocol,
host/domain and port. Basically, only send it when navigating through
our own UI, but not anything external.
The default is strict-origin-when-cross-origin when unset, which sends
the Referer header for requests unless it's going from HTTPS to HTTP
(i.e a security downgrade, hence the 'strict').
* [feature] Default to WASM-based SQLite driver
With 0.16 out this switches our default SQLite driver to the WASM-based
solution instead. So far the driver seems to perform just as well.
Switching our default should result in it getting a bit more testing
during the 0.17 development cycle.
* add the ol' john hancock
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* start updating media manager interface ready for storing attachments / emoji right away
* store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load()
* remove now unused media workers
* fix tests and issues
* fix another test!
* fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues
* fix more tests
* fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis
* whoops, rebase issue
* remove kim's whacky experiments
* do some reshuffling, ensure emoji uri gets set
* ensure marked as not cached on cleanup
* tweaks to media / emoji processing to handle context canceled better
* ensure newly fetched emojis actually get set in returned slice
* use different varnames to be a bit more obvious
* move emoji refresh rate limiting to dereferencer
* add exported dereferencer functions for remote media, use these for recaching in processor
* add check for nil attachment in updateAttachment()
* remove unused emoji and media fields + columns
* see previous commit
* fix old migrations expecting image_updated_at to exists (from copies of old models)
* remove freshness checking code (seems to be broken...)
* fix error arg causing nil ptr exception
* finish documentating functions with comments, slight tweaks to media / emoji deref error logic
* remove some extra unneeded boolean checking
* finish writing documentation (code comments) for exported media manager methods
* undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot
* move doesColumnExist() to util.go in migrations package
This syncs our copy with the current state of the ai.robots.txt
repository. Upstream has tightened their scope to be AI-only, whereas
before it included a bunch of SEO and "web intelligence" marketing
stuff. I've kept those but moved them into their own section.
* [bugfix] add Date and Message-ID headers for email
This should make spam filters more happy, as most of them grant some
negative score for not having those headers. Also the Date is convenient
for the user receiving the mail.
* make golangci-lint happy
* improved server shutdown with more precise shutdown of modules + deferring of ALL of it
* move delivery and workers into separate files
* add worker task model and Serialize() / Deserialize() methods for message types
* start adding message serialize / deserialize tests
* start adding test cases
* update body rewinding to rely on standard library mechanism of r.GetBody()
* remove request rewinding (http.Client{} should already handle this)
* standard library already handles rewinding
* improved code comment
* move the newPOST() function contents to prepare(), fits better with current API
* add Serialize() / Deserialize() implementations for Delivery{} type
* finish writing FromClientAPI sserialize / deserialize tests
* start adding FromFediAPI{} serialize / deserialize test cases
* fix FromFediAPI{} tests
* add tests for delivery message type
* fix repeat code
* missing license header
* use testrig status and accounts for marshal / unmarshaling tests
* add a specific test for checking account RSA keys are preserved
* use apiutil + paging in admin processor+handlers
* we're making it happen
* fix little whoopsie
* styling for report list
* don't youuuu forget about meee don't don't don't don't
* last bits
* sanitize content before showing in report statuses
* update report docs
These filter API v2 features were cut late in development because the form encoding version is hard to implement correctly and because I thought no clients actually used `keywords_attributes`. Unfortunately, Phanpy does use `keywords_attributes`.
* [bugfix] Deref stats async, allow peek if handshaking
* don't return totalItems when handshaking or hiding collections
* use GetLimit()
* use StubAccountStats
* fix possible infinite recursion if moved accounts are self-referential
* adds a defensive check for a boost being a boost of a boost wrapper
* add checks on input for a boost of a boost
* remove unnecessary check
* add protections on account move to prevent move recursion loops
* separate status conversion without boost logic into separate function to remove risk of recursion
* move boost check to boost function itself
* formatting
* use error 422 instead of 500
* use gtserror not standard errors package for error creation
* User muting
* Address review feedback
* Rename uniqueness constraint on user_mutes to match convention
* Remove unused account_id from where clause
* Add UserMute to NewTestDB
* Update test/envparsing.sh with new and fixed cache stuff
* Address tobi's review comments
* Make compiledUserMuteListEntry.expired consistent with UserMute.Expired
* Make sure mute_expires_at is serialized as an explicit null for indefinite mutes
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* refactor status media handling into separate functions, handle case of changed metadata
* update fetchRemoteAccount{Avatar,Header} to use new refactored {load,update}Attachment() functions
* whoops, nearly marked avatars as headers :')
* reformatting to improve legibility
* [feature] Email change
* frontend stuff for changing email
* docs
* tests etc
* differentiate more clearly between local user+account and account
* populate user
* do not uncache status / emoji media if attached status is bookmarked
* add status bookmark and bookmark IDs caches
* update status bookmark tests
* move IsStatusBookmarkedBy() to StatusBookmark{} interface, rely on cache
* fix envparsing.sh test
* removes the avatar / header deref maps as we now have per-uri status / account locks, adds retries on data-races, adds separate emoji map mutex
* work with a copy of account / status for each retry loop
* revert to old data race behaviour, it gets too complicated otherwise
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* Use correct entity name
* We support server-side filters now
* Document filter v1 methods that can throw a 409
* Validate v1 filter phrase as filter title
* Always check v1 filter API status codes in tests
* Document keyword minimum requirement on filter API v1
* Make it possible to specify filter keyword update columns per filter keyword
* Implement v2 filter API
* Fix lint and tests
* Update Swagger spec
* Fix filter update test
* Update Swagger spec *correctly*
* Update actual files Swagger spec was generated from
* Remove keywords_attributes and statuses_attributes
* Add test for serialization of empty filter
* More helpful messages when object is owned by wrong account
On startup and shutdown of a worker, we log a message of the worker
being started together with a textual representation of a memory
address. Though this can be handy for developers to debug
startup/shutdown sequencing issues of the workers, it's typically not
very useful or informative for an admin. We can also output a lot of
these (on my system I get 265 lines of these during startup).
This changes the messages from Info to Debug, to not print them under
normal circumstances.
* improved server shutdown with more precise shutdown of modules + deferring of ALL of it
* make the same changes to the testrig server
* use testrig specific func
* update variable name to fix nilptr
* fix removal of setting db on state
* Implement profile API
This Mastodon 4.2 extension provides capabilities missing from the existing Mastodon account update API: deleting an account's avatar or header.
See: https://docs.joinmastodon.org/methods/profile/
* Move profile media methods to media processor
* Remove check for moved account
* update to use go-storage/ instead of go-store/v2/storage/
* pull in latest version from codeberg
* remove test output 😇
* add code comments
* set the exclusive bit when creating new files in disk config
* bump to actual release version
* bump to v0.1.1 (tis a simple no-logic change)
* update readme
* only use a temporary read seeker when decoding video if required (should only be S3 now)
* use fastcopy library to use memory pooled buffers when calling TempFileSeeker()
* update to use seek call in serveFileRange()
* Remove dead code
* Filter statuses when converting to frontend representation
* status.filtered is an array
* Make matching case-insensitive
* Remove TODOs that don't need to be done now
* Add missing filter check for notification
* lint: rename ErrHideStatus
* APIFilterActionToFilterAction not used yet
* swaggerino docseroni
* Address review comments
* Add apimodel.FilterActionNone
---------
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* [feature] Page through accounts as moderator
* aaaaa
* use COLLATE "C" for Postgres to ensure same ordering as SQLite
* fix typo, test paging up
* don't show moderation / info for our instance acct
* start replacing client + federator + media workers with new worker + queue types
* refactor federatingDB.Delete(), drop queued messages when deleting account / status
* move all queue purging to the processor workers
* undo toolchain updates
* code comments, ensure dereferencer worker pool gets started
* update gruf libraries in readme
* start the job scheduler separately to the worker pools
* reshuffle ordering or server.go + remove duplicate worker start / stop
* update go-list version
* fix vendoring
* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped
* add logging to worker processing functions in testrig, don't start workers in unexpected places
* update go-structr to add (+then rely on) QueueCtx{} type
* ensure more worker pools get started properly in tests
* fix remaining broken tests relying on worker queue logic
* fix account test suite queue popping logic, ensure noop workers do not pull from queue
* move back accidentally shuffled account deletion order
* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()
* silently drop deletes from accounts not permitted to
* don't warn log on forwarded deletes
* make if else clauses easier to parse
* use getFederatorMsg()
* improved code comment
* improved code comment re: requesting account delete checks
* remove boolean result from worker start / stop since false = already running or already stopped
* remove optional passed-in http.client
* remove worker starting from the admin CLI commands (we don't need to handle side-effects)
* update prune cli to start scheduler but not all of the workers
* fix rebase issues
* remove redundant return statements
* i'm sorry sir linter
* [chore] Upgrade our Go version to 1.22
With Go 1.22 having been released at the start of February, it's now
been a few months. No major issues have shown up, and the two point
release since then have primarily been security fixes plus some general
bug fixing.
This sets the required Go version to 1.22, as there's nothing in 1.22.1
or 1.22.2 that we would explicitly require. It sets the toolchain to the
latest point release, to ensure we pick up any fixes from there when
building releases etc.
* [chore] Update CI to Go 1.22
* [chore] Update golangci-lint to 1.25.7
Newer version should know about Go 1.22 and run fine.
* [chore] Update Docker container to Go 1.22
* [chore] Update Dockerfile to newer Alpine version
* sign drone.yml
* add missing license header
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
This updates the robots.txt based on the list of the ai.robots.txt
repository. We can look at automating that at some point.
It's worth pointing out that some robots, namely the ones by Bytedance,
are known to ignore robots.txt entirely.
* [feature/performance] Store account stats in separate table, get stats from remote
* test account stats
* add some missing increment / decrement calls
* change stats function signatures
* rejig logging a bit
* use lock when updating stats
* update token + client code to use struct caches
* add code comments
* slight tweak to default mem ratios
* fix envparsing
* add appropriate invalidate hooks
* update the tokenstore sweeping function to rely on caches
* update to use PutClient()
* add ClientID to list of token struct indices
* update settings panels, add pending overview + approve/deny functions
* add admin accounts get, approve, reject
* send approved/rejected emails
* use signup URL
* docs!
* email
* swagger
* web linting
* fix email tests
* wee lil fixerinos
* use new paging logic for GetAccounts() series of admin endpoints, small changes to query building
* shuffle useAccountIDIn check *before* adding to query
* fix parse from toot react error
* use `netip.Addr`
* put valid slices in globals
* optimistic updates for account state
---------
Co-authored-by: kim <grufwub@gmail.com>
* [feature] User sign-up form and admin notifs
* add chosen + filtered languages to migration
* remove stray comment
* chosen languages schmosen schmanguages
* proper error on local account missing
* add delivery worker type that pulls from queue to httpclient package
* finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller
* hook up queue deletion logic
* support deleting queued http requests by target ID
* don't index APRequest by hostname in the queue
* use gorun
* use the original context's values when wrapping msg type as delivery{}
* actually log in the AP delivery worker ...
* add uncommitted changes
* use errors.AsV2()
* use errorsv2.AsV2()
* finish adding some code comments, add bad host handling to delivery workers
* slightly tweak deliveryworkerpool API, use advanced sender multiplier
* remove PopCtx() method, let others instead rely on Wait()
* shuffle things around to move delivery stuff into transport/ subpkg
* remove dead code
* formatting
* validate request before queueing for delivery
* finish adding code comments, fix up backoff code
* finish adding more code comments
* clamp minimum no. senders to 1
* add start/stop logging to delivery worker, some slight changes
* remove double logging
* use worker ptrs
* expose the embedded log fields in httpclient.Request{}
* ensure request context values are preserved when updating ctx
* add delivery worker tests
* fix linter issues
* ensure delivery worker gets inited in testrig
* fix tests to delivering messages to check worker delivery queue
* update error type to use ptr instead of value receiver
* fix test calling Workers{}.Start() instead of testrig.StartWorkers()
* update docs for advanced-sender-multiplier
* update to the latest activity library version
* add comment about not using httptest.Server{}
* [chore] Log less output on failed test
This changes the testrig log level to be error by default instead of
info. This makes test failures a lot easier to read, as we don't have
the parade of info logs for each failure to scroll through.
It speeds up the test suite by a couple of seconds since we need
to buffer and flush a lot less messages. On a clean run, so no test
failures, it's about a 3s difference on my machine. Depending on the
amount of test failures, total time saved can vary.
This also introduces a GTS_TESTRIG_LOG_LEVEL environment variable that
we explicitly check for, making it easy to override the log level should
we have a need for it. This would be primarily for running locally, and
not so much as part of go test.
Lastly, it updates the syslog tests to use log.Error because if the log
level is set to error but we call log.Info no message is emitted and we
hang indefinitely on the channel read.
* [chore] Rename the testrig log level env var
The id on the follows table is not a ULID, but a random ID. Sorting on
them results in a completely random order. Instead, sort on created_at,
which sould result in a stable and intended sort order.
Fixes: #2769
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
* improved PostInboxScheme() error handling / logging in case of failed auth
* dumbass kim. returning err instead of errWithCode...
* add checks for the slightly changed error handling in tests, add notes in codebase about the odd way of working
* [feature] User-selectable preset themes
* docs, more theme stuff
* lint, tests
* fix css name
* correct some little issues
* add another theme
* fix poll background
* okay last theme i swear
* make retrieval of apimodel themes more conventional
* preallocate stylesheet slices
tobi