* update activity library so dereferencer returns full response and checks *final* link to allow for redirects
* temporarily add bodged fixed library
* remove unused code
* update getAccountFeatured() to use dereferenceCollectionPage()
* make sure to release map
* perform a 2nd decode to ensure reader is empty after primary decode
* add comment explaining choice of using Decode() instead of Unmarshal()
* update embedded activity library to latest matching https://github.com/superseriousbusiness/activity/pull/21
* add checks to look for changed URI and re-check database if redirected
* update max iteration count to 512, add checks during dereferenceAncestors() for indirect URLs
* remove doubled-up code
* fix use of status instead of current
* use URIs for checking equality for security
* use the latest known URI for boost_of_uri in case original was an indirect
* add dereferenceCollection() function for dereferenceAccountFeatured()
* pull in latest github.com/superseriousbusiness/activity version (and remove the bodge!!)
* fix typo in code comments
* update decodeType() to accept a readcloser and handle body closing
* switch to checking using BoostOfID and add note why not using BoostOfURI
* ensure InReplyTo gets unset when deleting status parent in case currently stubbed
* add tests for Collection and CollectionPage iterators
* much simplified DereferenceStatusAncestors(), also handles edge cases now
* perform status acceptibility check before handling even as forward
* don't further dereference ancestors if they're up to date
* call enrichStatusSafely() directly to ensure we get error messages
* change getStatusByURI() semantics to return error + old model on failed update, fix deref ancestor to check for staleness before refetch
* perform a nil-check on the status.Local variable, in case it hasn't been set on new status attempting refresh
* more consistently set returned parent status, don't check if updated
* only home-timeline statuses if explicitly visible AND not explicitly invisible!
* fix broken test now that status acceptibility checks happen on forwarded statuses
* tidy up account, status, webfingering logic a wee bit
* go fmt
* invert published check
* alter resp initialization
* get Published from account in typeutils
* don't instantiate error for no darn good reason
* shadow err
* don't repeat error codes in wrapped errors
* don't wrap error unnecessarily
* only perform status-up-to-date checks if no statusable has been provided
* copy over the same style of freshness checking from status deref -> accounts
* change some var names
* check for empty account domain
* [bugfix] process account delete side effects in serial, not in parallel
* StartWorkers / StartNoopWorkers for tests
* undo testrig trace logging
* log errors instead of immediately returning
* update typeconverter to use state structure
* deinterface the typeutils.TypeConverter -> typeutils.Converter
* finish copying over old type converter code comments
* fix cherry-pick merge issues, fix tests pointing to old typeutils interface type still
* only attempt to populate account/statuses from DB if already up-to-date
Signed-off-by: kim <grufwub@gmail.com>
* add missing status is-up-to-date check :grimace: + ensure populated if so
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* media manager tidy-up: de-interface and remove unused PostDataFunc
Signed-off-by: kim <grufwub@gmail.com>
* remove last traces of media.Manager being an interface
Signed-off-by: kim <grufwub@gmail.com>
* update error to provide caller, allow tuneable via build tags
Signed-off-by: kim <grufwub@gmail.com>
* remove kim-specific build script changes
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts
Signed-off-by: kim <grufwub@gmail.com>
* update build-script to support externally setting build variables
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* revamp http client to not limit requests, instead use sender worker
Signed-off-by: kim <grufwub@gmail.com>
* remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping
Signed-off-by: kim <grufwub@gmail.com>
* improve batch senders to keep popping recipients until remote URL found
Signed-off-by: kim <grufwub@gmail.com>
* fix recipient looping issue
Signed-off-by: kim <grufwub@gmail.com>
* move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP
Signed-off-by: kim <grufwub@gmail.com>
* first draft of status refetching logic
Signed-off-by: kim <grufwub@gmail.com>
* fix testrig to use new federation alloc func signature
Signed-off-by: kim <grufwub@gmail.com>
* fix log format directive
Signed-off-by: kim <grufwub@gmail.com>
* add status fetched_at migration
Signed-off-by: kim <grufwub@gmail.com>
* remove unused / unchecked for error types
Signed-off-by: kim <grufwub@gmail.com>
* add back the used type...
Signed-off-by: kim <grufwub@gmail.com>
* add separate internal getStatus() function for derefThread() that doesn't recurse
Signed-off-by: kim <grufwub@gmail.com>
* improved mention and media attachment error handling
Signed-off-by: kim <grufwub@gmail.com>
* fix log and error format directives
Signed-off-by: kim <grufwub@gmail.com>
* update account deref to match status deref changes
Signed-off-by: kim <grufwub@gmail.com>
* very small code formatting change to make things clearer
Signed-off-by: kim <grufwub@gmail.com>
* add more code comments
Signed-off-by: kim <grufwub@gmail.com>
* improved code commenting
Signed-off-by: kim <grufwub@gmail.com>
* only check for required further derefs if needed
Signed-off-by: kim <grufwub@gmail.com>
* improved cache invalidation
Signed-off-by: kim <grufwub@gmail.com>
* tweak cache restarting to use a (very small) backoff
Signed-off-by: kim <grufwub@gmail.com>
* small readability changes and fixes
Signed-off-by: kim <grufwub@gmail.com>
* fix account sync issues
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts + update account enrichment to accept already-passed accountable
Signed-off-by: kim <grufwub@gmail.com>
* remove secondary function declaration
Signed-off-by: kim <grufwub@gmail.com>
* normalise dereferencer get status / account behaviour, fix remaining tests
Signed-off-by: kim <grufwub@gmail.com>
* fix remaining rebase conflicts, finish commenting code
Signed-off-by: kim <grufwub@gmail.com>
* appease the linter
Signed-off-by: kim <grufwub@gmail.com>
* add source file header
Signed-off-by: kim <grufwub@gmail.com>
* update to use TIMESTAMPTZ column type instead of just TIMESTAMP
Signed-off-by: kim <grufwub@gmail.com>
* don't pass in 'updated_at' to UpdateEmoji()
Signed-off-by: kim <grufwub@gmail.com>
* use new ap.Resolve{Account,Status}able() functions
Signed-off-by: kim <grufwub@gmail.com>
* remove the somewhat confusing rescoping of the same variable names
Signed-off-by: kim <grufwub@gmail.com>
* update migration file name, improved database delete error returns
Signed-off-by: kim <grufwub@gmail.com>
* formatting
Signed-off-by: kim <grufwub@gmail.com>
* improved multi-delete database functions to minimise DB calls
Signed-off-by: kim <grufwub@gmail.com>
* remove unused type
Signed-off-by: kim <grufwub@gmail.com>
* fix delete statements
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* [chore] Remove years from all license headers
Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.
In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.
* [chore] Add license header check
This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.
* [chore] Add missing license headers
* [chore] Further updates to license header
* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
* slight refactor and improved logging on failed webfinger in enrichAccount()
* use correct log format directive
---------
Signed-off-by: kim <grufwub@gmail.com>
* start fiddling
* the ol' fiddle + update
* start working on fetching statuses
* poopy doopy doo where r u uwu
* further adventures in featuring statuses
* finishing up
* fmt
* simply status unpin loop
* move empty featured check back to caller function
* remove unnecessary log.WithContext calls
* remove unnecessary IsIRI() checks
* add explanatory comment about status URIs
* change log level to error
* better test names
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.
Co-authored-by: kim <grufwub@gmail.com>
* start fiddling with media + account queries a little
* initialize state when pruning
* allow for unsetting remote media
make sure to wait til media loaded
fix silly tiny bug
* move comment a bit for readability
* slight reformat of fetchRemoteAccount{Avatar,Header}
* fix issue after rebase
* slightly neaten up logic of avatar/header media handling
* remove log prefix (callername log field handles this)
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* replace media workers with just runners.WorkerPool, move to state structure, use go-sched for global task scheduling
* improved code comment
* fix worker tryUntil function, update go-runners/go-sched
* make preprocess functions package public, use these where possible to stop doubled up processing
* remove separate emoji worker pool
* limit calls to time.Now() during media preprocessing
* use Processor{} to manage singular runtime of processing media
* ensure workers get started when media manager is used
* improved error setting in processing media, fix media test
* port changes from processingmedia to processing emoji
* finish code commenting
* finish code commenting and comment-out client API + federator worker pools until concurrency worker pools replaced
* linterrrrrrrrrrrrrrrr
---------
Signed-off-by: kim <grufwub@gmail.com>
* small formatting changes, rewrite fetchRemoteMedia to use separate funcs + use mutex lock correctly
* move url parsing before acquiring mutex locks
* use wrapped mutexes to allow safe unlocking. (previously i did a fucky and passed mutex by value...)
* remove unused code
* use consistent map keying for dereferencing headers/avatars
---------
Signed-off-by: kim <grufwub@gmail.com>
* reformat GetAccount() functionality, and add UpdateAccount() function.
* use fetched_at instead of last_webfingered_at
* catch local "not found" errors. small formatting / error string changes
* remove now unused error type
* return nil when wrapping nil error
* update expected error messages
* return correct url for foss satan webfinger
* add AP model for Some_User
* normalize local domain
* return notretrievable where appropriate
* expose NewErrNotRetrievable
* ensure webfinger for new accounts searched by uri
* update local account short circuit
* allow enrich to fail for already-known accounts
* remove unused LastWebfingeredAt
* expose test maps on mock http client
* update Update test
* reformat GetAccount() functionality, and add UpdateAccount() function.
* use fetched_at instead of last_webfingered_at
* catch local "not found" errors. small formatting / error string changes
* remove nil error checks (we shouldn't be passing nil errors to newError() initializers)
* remove mutex unlock on transport init fail (it hasn't yet been locked!)
* woops add back the error wrapping to use ErrNotRetrievable
* caches were never being started... 🙈
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* start refactoring some of the search + deref logic
* add tests for search api
* rename GetRemoteAccount + GetRemoteStatus
* make search function a bit simpler + clearer
* fix little fucky wucky uwu owo i'm just a little guy
* update faulty switch statements
* update test to use storage struct
* redo switches for clarity
* reduce repeated logic in search tests
* fastfail getstatus by uri
* debug log + trace log better
* add implementation note
* return early if no result for namestring search
* return + check on dereferencing error types
* errors hah what errors
* remove unneeded error type alias, add custom error text during stringification itself
* fix a woops recursion 🙈
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* [feature/performance] Fail fast when doing remote transport calls inside incoming request contexts
* [chore] Reduce outgoing request timeout to 15s
* log error messages when fastfailing
* use context.Value() instead of wrapped context, wrap error with fastfail instead of extra log entry
* add fast-fail context key test
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* convert most of the caches to use result.Cache{}
* add caching of emojis
* fix issues causing failing tests
* update go-cache/v2 instances with v3
* fix getnotification
* add a note about the left-in StatusCreate comment
* update EmojiCategory db access to use new result.Cache{}
* fix possible panic in getstatusparents
* further proof that kim is not stinky
* fix incorrect static remote url use for emojis
* warn when emoji/attachment already exists
* defer emoji postdata execution
* rename ctx to innerCtx for clarity
* warn on emoji too large
* small efficiency fix in fetchRemoteAccountEmojis
* tidy up lock+load
* lock processing emojis
* fix little fucky wucky
* this wasn't go fmted for some reason
* use readcloser for content.Content
* call media postdata function no matter what
* return a readcloser from data func
* tidy of logic of readertostore
* fix whoopsie
* Start adding account emoji
* get emojis serialized + deserialized nicely
* update tests
* set / retrieve emojis on accounts
* show account emojis in web view
* fetch emojis from db based on ids
* fix typo in test
* lint
* fix pg migration
* update tests
* update emoji checking logic
* update comment
* clarify comments + add some spacing
* tidy up loops a lil (thanks kim)
* update Activity
* add instance-deliver-to-shared-inboxes setting
* update activity version again
* add SharedInboxURI field to accounts
* serdes for endpoints/sharedInbox
* deliver to sharedInbox if one is available
* update tests
* only assign shared inbox if shared domain
* look for shared inbox if currently nil
* go fmt
* finger to get params.RemoteAccountID if necessary
* make comments clearer
* compare dns more consistently
* first commit
Signed-off-by: kim <grufwub@gmail.com>
* replace logging with our own log library
Signed-off-by: kim <grufwub@gmail.com>
* fix imports
Signed-off-by: kim <grufwub@gmail.com>
* fix log imports
Signed-off-by: kim <grufwub@gmail.com>
* add license text
Signed-off-by: kim <grufwub@gmail.com>
* fix package import cycle between config and log package
Signed-off-by: kim <grufwub@gmail.com>
* fix empty kv.Fields{} being passed to WithFields()
Signed-off-by: kim <grufwub@gmail.com>
* fix uses of log.WithFields() with whitespace issues and empty slices
Signed-off-by: kim <grufwub@gmail.com>
* *linter related grumbling*
Signed-off-by: kim <grufwub@gmail.com>
* gofmt the codebase! also fix more log.WithFields() formatting issues
Signed-off-by: kim <grufwub@gmail.com>
* update testrig code to match new changes
Signed-off-by: kim <grufwub@gmail.com>
* fix error wrapping in non fmt.Errorf function
Signed-off-by: kim <grufwub@gmail.com>
* add benchmarking of log.Caller() vs non-cached
Signed-off-by: kim <grufwub@gmail.com>
* fix syslog tests, add standard build tags to test runner to ensure consistency
Signed-off-by: kim <grufwub@gmail.com>
* make syslog tests more robust
Signed-off-by: kim <grufwub@gmail.com>
* fix caller depth arithmatic (is that how you spell it?)
Signed-off-by: kim <grufwub@gmail.com>
* update to use unkeyed fields in kv.Field{} instances
Signed-off-by: kim <grufwub@gmail.com>
* update go-kv library
Signed-off-by: kim <grufwub@gmail.com>
* update libraries list
Signed-off-by: kim <grufwub@gmail.com>
* fuck you linter get nerfed
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
* move finger to dereferencer
* totally break GetRemoteAccount
* start reworking finger func a bit
* start reworking getRemoteAccount a bit
* move mention parts to namestring
* rework webfingerget
* use util function to extract webfinger parts
* use accountDomain
* rework finger again, final form
* just a real nasty commit, the worst
* remove refresh from account
* use new ASRepToAccount signature
* fix incorrect debug call
* fix for new getRemoteAccount
* rework GetRemoteAccount
* start updating tests to remove repetition
* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).
* fix up tests to use main mock httpclient
* webfinger only if necessary
* cheeky linting with the lads
* update mentionName regex
recognize instance accounts
* don't finger instance accounts
* test webfinger part extraction
* increase default worker count to 4 per cpu
* don't repeat regex parsing
* final search for discovered accountDomain
* be more permissive in namestring lookup
* add more extraction tests
* simplify GetParseMentionFunc
* skip long search if local account
* fix broken test
kim