mirror of
1
Fork 0
Commit Graph

84 Commits

Author SHA1 Message Date
kim c9d36f7e45
[performance] use new instance of bun.DB *after* migrations to reduce number of in-memory model schema (#3578)
* use new instance of bun.DB *after* migrations to reduce number of model schema in-memory

* update sqlite address comment
2024-11-29 15:03:10 +00:00
tobi c454b1b488
[chore] Bump tooling versions, bump go -> v1.23.0 (#3258)
* [chore] Bump tooling versions, bump go -> v1.23.0

* undo silly change

* sign

* bump go version in go.mod

* allow overflow in imaging

* goreleaser deprecation notices

* bump versions

* undo accidental rebase change

* update container versions to just use latest major version

* update swagger to our release with go1.23 fix

* update goreleaser to use our vendored swagger version

---------

Co-authored-by: kim <grufwub@gmail.com>
2024-11-25 16:15:33 +01:00
Markus Unterwaditzer a48cce82b9
[chore] Upgrade golangci-lint, ignore existing int overflow warnings (#3420)
* [chore] Bump tooling versions, bump go -> v1.23.0

* undo silly change

* sign

* bump go version in go.mod

* allow overflow in imaging

* goreleaser deprecation notices

* [chore] Upgrade golangci-lint, ignore existing int overflow warnings

There is a new lint for unchecked int casts. Integer overflows are bad,
but the old code that triggers this lint seems to be perfectly fine.
Instead of disabling the lint entirely for new code as well, grandfather
in existing code.

* fix golangci-lint documentation link

* revert unrelated changes

* revert another unrelated change

* get rid of remaining nolint:gosec

* swagger updates

* apply review feedback

* fix wrong formatting specifier thing

* fix the linter for real

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-10-16 14:13:58 +02:00
kim 2076f7d85f
[feature] for an sqlite database with journal mode != WAL, use maximum of 1 open conn (#3428) 2024-10-14 00:19:52 +02:00
tobi 307d98e386
[feature] Process `Reject` of interaction via fedi API, put rejected statuses in the "sin bin" 😈 (#3271)
* [feature] Process `Reject` of interaction via fedi API, put rejected statuses in the "sin bin"

* update test

* move nil check back to `rejectStatusIRI`
2024-09-10 12:34:49 +00:00
tobi b17010cf17
[bugfix] Fix temp table deletion causing runaway allocations (#3278)
* [bugfix] Fix temp table deletion causing runaway allocations

* move some vars around

* small fixes

* rely on conn max age to recycle temp tables

* fackin' ell m8
2024-09-08 16:14:56 +02:00
John Winston 94c615d417
[feature] Add `db-postgres-connection-string` option (#3178)
* handle db-url

* lint and add doc

* add more doc

* fix config test

* return error

* change name from db-url to db-postgres-connection-string
2024-08-08 14:00:19 +02:00
kim 87cff71af9
[feature] persist worker queues to db (#3042)
* persist queued worker tasks to database on shutdown, fill worker queues from database on startup

* ensure the tasks are sorted by creation time before pushing them

* add migration to insert WorkerTask{} into database, add test for worker task persistence

* add test for recovering worker queues from database

* quick tweak

* whoops we ended up with double cleaner job scheduling

* insert each task separately, because bun is throwing some reflection error??

* add specific checking of cancelled worker contexts

* add http request signing to deliveries recovered from database

* add test for outgoing public key ID being correctly set on delivery

* replace select with Queue.PopCtx()

* get rid of loop now we don't use it

* remove field now we don't use it

* ensure that signing func is set

* header values weren't being copied over 🤦

* use ptr for httpclient.Request in delivery

* move worker queue filling to later in server init process

* fix rebase issues

* make logging less shouty

* use slices.Delete() instead of copying / reslicing

* have database return tasks in ascending order instead of sorting them

* add a 1 minute timeout to persisting worker queues
2024-07-30 13:58:31 +02:00
Vyr Cossont 8fdd358f4b
[feature] Conversations API (#3013)
* Implement conversations API

* Sort and page conversations by last status ID

* Appease linter

* Fix deleting conversations and statuses

* Refactor to make migrations automatic

* Lint

* Update tests post-merge

* Fixes from live-fire testing

* Linter caught a format problem

* Refactor tests, fix cache

* Negative test for non-DMs

* Run conversations advanced migration on testrig startup as well as regular server startup

* Document (lack of) side effects of API method for deleting a conversation

* Make not-found check less nested for readability

* Rename PutConversation to UpsertConversation

* Use util.Ptr instead of IIFE

* Reduce cache used by conversations

* Remove unnecessary TableExpr/ColumnExpr

* Use struct tags for both unique constraints on Conversation

* Make it clear how paging with GetDirectStatusIDsBatch should be used

* Let conversation paging skip conversations it can't render

* Use Bun NewDropTable

* Convert delete raw query to Bun

* Convert update raw query to Bun

* Convert latestConversationStatusesTempTable raw query partially to Bun

* Convert conversationStatusesTempTable raw query partially to Bun

* Rename field used to store result of MaxDirectStatusID

* Move advanced migrations to their own tiny processor

* Catch up util function name with main

* Remove json.… wrappers

* Remove redundant check

* Combine error checks

* Replace map with slice of structs

* Address processor/type converter comments

- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation

* Add error context when dropping temp tables
2024-07-23 20:44:31 +01:00
tobi 5bc567196b
[chore] Add interaction policy gtsmodels (#3075)
* [chore] introduce interaction policy gts models

* update migration a smidge

* fix copy paste typo

* update migration

* use int for InteractionType
2024-07-11 15:44:29 +01:00
kim 1e7b32490d
[experiment] add alternative wasm sqlite3 implementation available via build-tag (#2863)
This allows for building GoToSocial with [SQLite transpiled to WASM](https://github.com/ncruces/go-sqlite3) and accessed through [Wazero](https://wazero.io/).
2024-05-27 17:46:15 +02:00
tobi 9b94b1faa2
[docs] Add database maintenance section; update info message on ANALYZE run (sqlite) (#2756)
* [chore] Limit size of ANALYZE run after migration (sqlite)

* add basic db maintenance tips

* update docs, analyze

* amend info message a wee bit

* update docs/admin/database_maintenance.md wording

Co-authored-by: Daenney <daenney@users.noreply.github.com>

---------

Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
Co-authored-by: Daenney <daenney@users.noreply.github.com>
2024-03-14 17:40:36 +00:00
tobi b22e213e15
[feature/chore] Add Move database functions + cache (#2647)
* [feature/chore] Add Move database functions + cache

* add move mem ratio to envparsing.sh

* update comment
2024-03-06 11:18:57 +01:00
Vyr Cossont 61a2b91f45
[feature] Filters v1 (#2594)
* Implement client-side v1 filters

* Exclude linter false positives

* Update test/envparsing.sh

* Fix minor Swagger, style, and Bun usage issues

* Regenerate Swagger

* De-generify filter keywords

* Remove updating filter statuses

This is an operation that the Mastodon v2 filter API doesn't actually have, because filter statuses, unlike keywords, don't have options: the only info they contain is the status ID to be filtered.

* Add a test for filter statuses specifically

* De-generify filter statuses

* Inline FilterEntry

* Use vertical style for Bun operations consistently

* Add comment on Filter DB interface

* Remove GoLand linter control comments

Our existing linters should catch these, or they don't matter very much

* Reduce memory ratio for filters
2024-03-06 11:15:58 +01:00
kim 6738fd5bb0
[feature/performance] sqlite pragma optimize on close (#2596)
* wrap database drivers in order to handle error processing, hooks, etc

* remove dead code

* add code comment, remove unused blank imports
2024-02-07 14:43:27 +00:00
kim 7ec1e1332e
[performance] overhaul struct (+ result) caching library for simplicity, performance and multiple-result lookups (#2535)
* rewrite cache library as codeberg.org/gruf/go-structr, implement in gotosocial

* use actual go-structr release version (not just commit hash)

* revert go toolchain changes (damn you go for auto changing this)

* fix go mod woes

* ensure %w is used in calls to errs.Appendf()

* fix error checking

* fix possible panic

* remove unnecessary start/stop functions, move to main Cache{} struct, add note regarding which caches require start/stop

* fix copy-paste artifact... 😇

* fix all comment copy-paste artifacts

* remove dropID() function, now we can just use slices.DeleteFunc()

* use util.Deduplicate() instead of collate(), move collate to util

* move orderByIDs() to util package and "generify"

* add a util.DeleteIf() function, use this to delete entries on failed population

* use slices.DeleteFunc() instead of util.DeleteIf() (i had the logic mixed up in my head somehow lol)

* add note about how collate differs from deduplicate
2024-01-19 12:57:29 +00:00
kim 8ebb7775a3
[feature] request blocking by http headers (#2409) 2023-12-18 14:18:25 +00:00
Daenney fbe4e60232
[feature] Run ANALYZE after migrations on SQLite (#2428)
* [feature] Run ANALYZE after migrations on SQLite

This ensures that at the end of migrations, we run ANALYZE if we're
using SQLite. This should be relatively quick and guarantees that the
table and index statistics have been updated. This helps to ensure the
query planner makes better choices when it comes to picking which
indexes are used when running queries.

* [chore] use ExecContext

Uses ExecContext so we pass the context through, this is helpful for
anyone running with tracing enabled
2023-12-16 12:54:53 +01:00
Tsuribori 1ba3e14b36
[feature] Initial Prometheus metrics implementation (#2334)
* feat: Initial OTEL metrics

* docs: add metrics documentation

* fix: metrics endpoint conditional check

* feat: metrics endpoint basic auth

* fix: make metrics-auth-enabled default false

* fix: go fmt helpers.gen.go

* fix: add metric-related env vars to envparsing.sh

* fix: metrics docs

* fix: metrics related stuff in envparsing.sh

* fix: metrics docs

* chore: metrics docs wording

* fix: metrics stuff in envparsing?

* bump otel versions

---------

Co-authored-by: Tsuribori <user@acertaindebian>
Co-authored-by: Tsuribori <none@example.org>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-11-20 16:43:55 +01:00
kim e9e5dc5a40
[feature] add support for polls + receiving federated status edits (#2330) 2023-11-08 14:32:17 +00:00
tobi c7b6cd7770
[feature] Status thread mute/unmute functionality (#2278)
* add db models + functions for keeping track of threads

* give em the old linty testy

* create, remove, check mutes

* swagger

* testerino

* test mute/unmute via api

* add info log about new index creation

* thread + allow muting of any remote statuses that mention a local account

* IsStatusThreadMutedBy -> IsThreadMutedByAccount

* use common processing functions in status processor

* set = NULL

* favee!

* get rekt darlings, darlings get rekt

* testrig please, have mercy muy liege
2023-10-25 15:04:53 +01:00
tobi 013a17f5e8
[chore] bump go version -> 1.21.x (#2287)
* [chore] bump go version -> 1.21.x

* versions

* lint
2023-10-23 13:07:31 +01:00
kim 4eb77ff5d7
[bugfix] move SQLite pragmas into connection string (#2171)
* move SQLite pragmas into connection string

Signed-off-by: kim <grufwub@gmail.com>

* use url.Values type for SQLite connection preferences

Signed-off-by: kim <grufwub@gmail.com>

* set SQLite URI prefs properly using _pragma query key

Signed-off-by: kim <grufwub@gmail.com>

* add notes on SQLite connection preferences

Signed-off-by: kim <grufwub@gmail.com>

* fix typo

Signed-off-by: kim <grufwub@gmail.com>

* add one extra line regarding connection pooling

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-09-01 15:13:33 +02:00
f0x52 92de8fb396
[feature] Instance rules (#2125)
* init instance rules database model, admin api

* expose instance rules in public instance api

* public /api/v1/instance/rules route

* GET ruleById

* createRule route

* createRule auth check

* updateRule

* deleteRule

* list rules on about page

* ruleGet auth

* add about page ids for anchors

* process and store adding violated rules to reports

* admin api models for instance rules

* instance rule edit frontend

* change rule inputs to textareas

* database fixes after rebase (#2124)

* remove unused imports

* fix db migration column name

* fix tests

* fix more tests

* fix postgres error with wrongly used Ident

* add some tests, fiddle with rule model a bit, fix postgres migration

* swagger docs

---------

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-08-19 14:33:15 +02:00
kim d5d6ad406f
[bugfix] fix double firing bun.DB query hooks (#2124)
* improve bun.DB wrapping readability + comments, fix double-firing query hooks

* fix incorrect code comment placement

* fix linter issues

* Update internal/db/basic.go

* do as the linter commmands ...

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: Daenney <daenney@users.noreply.github.com>
2023-08-17 17:26:21 +01:00
kim 91cbcd589e
[performance] remove last of relational queries to instead rely on caches (#2091) 2023-08-10 15:08:41 +01:00
tobi 2796a2e82f
[feature] Hashtag federation (in/out), hashtag client API endpoints (#2032)
* update go-fed

* do the things

* remove unused columns from tags

* update to latest lingo from main

* further tag shenanigans

* serve stub page at tag endpoint

* we did it lads

* tests, oh tests, ohhh tests, oh tests (doo doo doo doo)

* swagger docs

* document hashtag usage + federation

* instanceGet

* don't bother parsing tag href

* rename whereStartsWith -> whereStartsLike

* remove GetOrCreateTag

* dont cache status tag timelineability
2023-07-31 15:47:35 +02:00
Vyr Cossont b874e9251e
[feature] Implement markers API (#1989)
* Implement markers API

Fixes #1856

* Correct import grouping in markers files

* Regenerate Swagger for markers API

* Shorten names for readability

* Cache markers for 6 hours

* Update DB ref

* Update envparsing.sh
2023-07-29 12:49:14 +02:00
kim 5f3e095717
[performance] retry db queries on busy errors (#2025)
* catch SQLITE_BUSY errors, wrap bun.DB to use our own busy retrier, remove unnecessary db.Error type

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* remove more dead code, add missing error arguments

Signed-off-by: kim <grufwub@gmail.com>

* update sqlite to use maxOpenConns()

Signed-off-by: kim <grufwub@gmail.com>

* add uncommitted changes

Signed-off-by: kim <grufwub@gmail.com>

* use direct calls-through for the ConnIface to make sure we don't double query hook

Signed-off-by: kim <grufwub@gmail.com>

* expose underlying bun.DB better

Signed-off-by: kim <grufwub@gmail.com>

* retry on the correct busy error

Signed-off-by: kim <grufwub@gmail.com>

* use longer possible maxRetries for db retry-backoff

Signed-off-by: kim <grufwub@gmail.com>

* remove the note regarding max-open-conns only applying to postgres

Signed-off-by: kim <grufwub@gmail.com>

* improved code commenting

Signed-off-by: kim <grufwub@gmail.com>

* remove unnecessary infof call (just use info)

Signed-off-by: kim <grufwub@gmail.com>

* rename DBConn to WrappedDB to better follow sql package name conventions

Signed-off-by: kim <grufwub@gmail.com>

* update test error string checks

Signed-off-by: kim <grufwub@gmail.com>

* shush linter

Signed-off-by: kim <grufwub@gmail.com>

* update backoff logic to be more transparent

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-07-25 10:34:05 +02:00
tobi e70bf8a6c8
[chore/bugfix] Domain block tidying up, Implement first pass of `207 Multi-Status` (#1886)
* [chore/refactor] update domain block processing

* expose domain block import errors a lil better

* move/remove unused query keys
2023-07-07 11:34:12 +02:00
tobi 831ae09f8b
[feature] Add partial text search for accounts + statuses (#1836) 2023-06-21 17:26:40 +01:00
tobi f5c004d67d
[feature] Add List functionality (#1802)
* start working on lists

* further list work

* test list db functions nicely

* more work on lists

* peepoopeepoo

* poke

* start list timeline func

* we're getting there lads

* couldn't be me working on stuff... could it?

* hook up handlers

* fiddling

* weeee

* woah

* screaming, pissing

* fix streaming being a whiny baby

* lint, small test fix, swagger

* tidying up, testing

* fucked! by the linter

* move timelines to state like a boss

* add timeline start to tests using state

* invalidate lists
2023-05-25 10:37:38 +02:00
tobi ec325fee14
[chore] Update a bunch of database dependencies (#1772)
* [chore] Update a bunch of database dependencies

* fix lil thing
2023-05-12 14:33:40 +02:00
Dominik Süß 6392e00653
feat: initial tracing support (#1623) 2023-05-09 18:19:48 +01:00
tobi e8595f0c64
[chore] Refactor account deleting/block logic, tidy up some other processing things (#1599)
* start refactoring account deletion

* update to use state.DB

* further messing about

* some more tidying up

* more tidying, cleaning, nice-making

* further adventures in refactoring and the woes of technical debt

* update fr accept/reject

* poking + prodding

* fix up deleting

* create fave uri

* don't log using requestingAccount.ID because it might be nil

* move getBookmarks function

* use exists query to check for status bookmark

* use deletenotifications func

* fiddle

* delete follow request notif

* split up some db functions

* Fix possible nil pointer panic

* fix more possible nil pointers

* fix license headers

* warn when follow missing (target) account

* return wrapped err when bookmark/fave models can't be retrieved

* simplify self account delete

* warn log likely race condition

* de-sillify status delete loop

* move error check due north

* warn when unfollowSideEffects has no target account

* warn when no boost account is found

* warn + dump follow when no account

* more warnings

* warn on fave account not set

* move for loop inside anonymous function

* fix funky logic

* don't remove mutual account items on block;
do make sure unfollow occurs in both directions!
2023-03-20 18:10:08 +00:00
Daenney 5e2bf0bdca
[chore] Improve copyright header handling (#1608)
* [chore] Remove years from all license headers

Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.

In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.

* [chore] Add license header check

This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.

* [chore] Add missing license headers

* [chore] Further updates to license header

* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
2023-03-12 16:00:57 +01:00
Daenney 68e6d08c76
[feature] Add a request ID and include it in logs (#1476)
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.

Co-authored-by: kim <grufwub@gmail.com>
2023-02-17 12:02:29 +01:00
tobi 561ad71e58
[bugfix] Fix up `error getting account avatar/header` errors, other small fixes (#1496)
* start fiddling with media + account queries a little

* initialize state when pruning

* allow for unsetting remote media
make sure to wait til media loaded
fix silly tiny bug

* move comment a bit for readability

* slight reformat of fetchRemoteAccount{Avatar,Header}

* fix issue after rebase

* slightly neaten up logic of avatar/header media handling

* remove log prefix (callername log field handles this)

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2023-02-13 20:19:51 +00:00
Autumn! 49beb17a8f
[chore] Text formatting overhaul (#1406)
* Implement goldmark debug print for hashtags and mentions

* Minify HTML in FromPlain

* Convert plaintext status parser to goldmark

* Move mention/tag/emoji finding logic into formatter

* Combine mention and hashtag boundary characters

* Normalize unicode when rendering hashtags
2023-02-03 11:58:58 +01:00
tobi 4ee4cd2da1
[chore/performance] use only 1 sqlite db connection regardless of multiplier (#1408) 2023-02-01 11:55:34 +01:00
tobi b80be48fed
[chore] Use 'immediate' lock for sqlite transactions (#1404)
* [chore] Use 'immediate' lock for sqlite transactions

* allow 1 connection regardless of cpu amount
2023-01-31 13:46:45 +01:00
tobi 782169da76
[chore] set max open / idle conns + conn max lifetime for both postgres and sqlite (#1369)
* [chore] set max open / idle conns + conn max lifetime for both postgres and sqlite

* reduce cache size default to 8MiB, reduce connections to 2 * cpu

* introduce max open conns multiplier, tune sqlite and pg separately

* go fmt
2023-01-26 15:12:48 +01:00
tobi 627b8eeae6
[feature] Tune sqlite pragmas (#1349)
* sqlite pragma tuning

* use formatuint

* add sqlite busy timeout

* fix incorrect cache size format

* update envparsing test

* add sqlite tuning flags to cli

* set sqlite timeout to 30s default
2023-01-17 12:29:44 +00:00
tobi d6487933c7
[feature] Implement Report database model and utility functions (#1310)
* implement report database model

* implement report cache + config changes

* implement report database functions

* report uri / regex functions

* update envparsing test

* remove unnecessary uri index

* remove unused function + cache lookup

* process error when storing report
2023-01-10 14:19:05 +00:00
tobi 0dbe6c514f
[chore] Update/add license headers for 2023 (#1304) 2023-01-05 12:43:00 +01:00
kim e58d2d8122
[chore] move caches to a separate State{} structure (#1078)
* move caches to a separate State{} structure

Signed-off-by: kim <grufwub@gmail.com>

* fix call to log.Panic not using formatted call

Signed-off-by: kim <grufwub@gmail.com>

* move caches to use interfaces, to make switchouts easier in future

Signed-off-by: kim <grufwub@gmail.com>

* fix rebase issue

Signed-off-by: kim <grufwub@gmail.com>

* improve code comment

Signed-off-by: kim <grufwub@gmail.com>

* fix further issues after rebase

Signed-off-by: kim <grufwub@gmail.com>

* heh

Signed-off-by: kim <grufwub@gmail.com>

* add missing license text

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-12-08 18:35:14 +01:00
kim 5d55e8d920
[performance] add account block DB cache and remove block query joins (#1085)
* add account block DB cache and remove reliance on relational joins

* actually include cache key arguments...

* add a PutBlock() method which also updates the block cache, update tests accordingly

* use `PutBlock` instead of `Put(ctx, block)`

* add + use functions for deleting + invalidating blocks

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-11-20 16:33:49 +00:00
kim 45ae719bd9
[performance] replace account emojis relational query with separate calls to emojiDB to rely on cache (#1074)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-18 18:29:25 +01:00
kim dccc2eee81
[performance] replace status query relationals with separate calls in order to rely on caches more (#1073)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-18 18:28:33 +01:00
kim 8598dea98b
[chore] update database caching library (#1040)
* convert most of the caches to use result.Cache{}

* add caching of emojis

* fix issues causing failing tests

* update go-cache/v2 instances with v3

* fix getnotification

* add a note about the left-in StatusCreate comment

* update EmojiCategory db access to use new result.Cache{}

* fix possible panic in getstatusparents

* further proof that kim is not stinky
2022-11-15 18:45:15 +00:00