* feat: vendor minio client
* feat: introduce storage package with s3 support
* feat: serve s3 files directly
this saves a lot of bandwith as the files are fetched from the object
store directly
* fix: use explicit local storage in tests
* feat: integrate s3 storage with the main server
* fix: add s3 config to cli tests
* docs: explicitly set values in example config
also adds license header to the storage package
* fix: use better http status code on s3 redirect
HTTP 302 Found is the best fit, as it signifies that the resource
requested was found but not under its presumed URL
307/TemporaryRedirect would mean that this resource is usually located
here, not in this case
303/SeeOther indicates that the redirection does not link to the
requested resource but to another page
* refactor: use context in storage driver interface
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct
Signed-off-by: kim <grufwub@gmail.com>
* improved code comment
Signed-off-by: kim <grufwub@gmail.com>
* linter
Signed-off-by: kim <grufwub@gmail.com>
* fix unmarshaling
Signed-off-by: kim <grufwub@gmail.com>
* remove kim's custom go compiler changes
Signed-off-by: kim <grufwub@gmail.com>
* generate setter and flag-name functions, implement these in codebase
Signed-off-by: kim <grufwub@gmail.com>
* update deps
Signed-off-by: kim <grufwub@gmail.com>
* small change
Signed-off-by: kim <grufwub@gmail.com>
* appease the linter...
Signed-off-by: kim <grufwub@gmail.com>
* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync
Signed-off-by: kim <grufwub@gmail.com>
* lint
Signed-off-by: kim <grufwub@gmail.com>
* update code comments
Signed-off-by: kim <grufwub@gmail.com>
* fix merge issue
Signed-off-by: kim <grufwub@gmail.com>
* fix merge issue
Signed-off-by: kim <grufwub@gmail.com>
* improved version string (removes time + go version)
Signed-off-by: kim <grufwub@gmail.com>
* fix version string build to pass test script + consolidate logic in func
Signed-off-by: kim <grufwub@gmail.com>
* add license text, update config.Defaults comment
Signed-off-by: kim <grufwub@gmail.com>
* add license text to generated config helpers file
Signed-off-by: kim <grufwub@gmail.com>
* defer unlock on config.Set___(), to ensure unlocked on panic
Signed-off-by: kim <grufwub@gmail.com>
* make it more obvious which cmd flags are being attached
Signed-off-by: kim <grufwub@gmail.com>
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests
Signed-off-by: kim <grufwub@gmail.com>
* update code comments, defer mutex unlocks
Signed-off-by: kim <grufwub@gmail.com>
* add count to 'performing request' log message
Signed-off-by: kim <grufwub@gmail.com>
* reduce repeated conversions of same url.URL object
Signed-off-by: kim <grufwub@gmail.com>
* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue
Signed-off-by: kim <grufwub@gmail.com>
* fix security advisories regarding max outgoing conns, max rsp body size
- implemented by a new httpclient.Client{} that wraps an underlying
client with a queue to limit connections, and limit reader wrapping
a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}
Signed-off-by: kim <grufwub@gmail.com>
* add httpclient tests, move ip validation to separate package + change mechanism
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts
Signed-off-by: kim <grufwub@gmail.com>
* use singular mutex in transport rather than separate signer mus
Signed-off-by: kim <grufwub@gmail.com>
* improved useragent string
Signed-off-by: kim <grufwub@gmail.com>
* add note regarding missing test
Signed-off-by: kim <grufwub@gmail.com>
* remove useragent field from transport (instead store in controller)
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* reset other signing headers on each loop iteration
Signed-off-by: kim <grufwub@gmail.com>
* respect request ctx during retry-backoff sleep period
Signed-off-by: kim <grufwub@gmail.com>
* use external pkg with docs explaining performance "hack"
Signed-off-by: kim <grufwub@gmail.com>
* use http package constants instead of string method literals
Signed-off-by: kim <grufwub@gmail.com>
* add license file headers
Signed-off-by: kim <grufwub@gmail.com>
* update code comment to match new func names
Signed-off-by: kim <grufwub@gmail.com>
* updates to user-agent string
Signed-off-by: kim <grufwub@gmail.com>
* update signed testrig models to fit with new transport logic (instead uses separate signer now)
Signed-off-by: kim <grufwub@gmail.com>
* fuck you linter
Signed-off-by: kim <grufwub@gmail.com>
* fix error check of prune to allow missing files
* update go-store library, add test for pruning item with db entry but no file
Signed-off-by: kim <grufwub@gmail.com>
* remove now-unneccessary error check
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* wrap root HTTP handler in debug.WithPprof(), rearrange router.Start() to support this
* remove unused code
* set debug buildtag in build script when $DEBUG set
* update go-debug version with fixed handler
* use clone of router.srv for LE cert manager, reset server timeouts in debug
* add kim's other libraries to README
* linting with new golangci-lint version
* update go to 1.18
* bump versions in drone.yml
* use new runtime/debug package for version info
* remove Commit build flag from goreleaser
* remove mock commit + version from build script
* go fmt
* add dummy version env flag to test container
* install git in golang container for testing
* only set versionString if Version is defined
* Add whereNotEmptyAndNotNull
* Add GetRemoteOlderThanDays
* Add GetRemoteOlderThanDays
* Add PruneRemote to Manager interface
* Start implementing PruneRemote
* add new attachment + status to tests
* fix up and test GetRemoteOlderThan
* fix bad import
* PruneRemote: return number pruned
* add Cached column to mediaattachment
* update + test pruneRemote
* update mediaTest
* use Cached column
* upstep bun to latest version
* embed structs in mediaAttachment
* migrate mediaAttachment to new format
* don't default cached to true
* select only remote media
* update db dependencies
* step bun back to last working version
* update pruneRemote to use Cached field
* fix storage path of test attachments
* add recache logic to manager
* fix trimmed aspect ratio
* test prune and recache
* return errwithcode
* tidy up different paths for emoji vs attachment
* fix incorrect thumbnail type being stored
* expose TransportController to media processor
* implement tee-ing recached content
* add thoughts of dog to test fedi attachments
* test get remote files
* add comment on PruneRemote
* add postData cleanup to recache
* test thumbnail fetching
* add incredible diagram
* go mod tidy
* buffer pipes for recache streaming
* test for client stops reading after 1kb
* add media-remote-cache-days to config
* add cron package
* wrap logrus so it's available to cron
* start and stop cron jobs gracefully
* start pulling out + replacing urfave and config
* replace many many instances of config
* move more stuff => viper
* properly remove urfave
* move some flags to root command
* add testrig commands to root
* alias config file keys
* start adding cli parsing tests
* reorder viper init
* remove config path alias
* fmt
* change config file keys to non-nested
* we're more or less in business now
* tidy up the common func
* go fmt
* get tests passing again
* add note about the cliparsing tests
* reorganize
* update docs with changes
* structure cmd dir better
* rename + move some files around
* fix dangling comma
* update bun library to latest commit
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* update to latest bun release
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* move scripts, allow testing both dbs with one cmd
* tidy + vendor
* update test.sh to ignore cache
* put test commands directly in drone.yml
* change CONTRIBUTING slightly
* go ham on the readme
* Add SQLite support, fix un-thread-safe DB caches, small performance fixes
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add SQLite licenses to README
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* appease the linter, and fix my dumbass-ery
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* make requested changes
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add back comment
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* start moving some database stuff around
* continue moving db stuff around
* more fiddling
* more updates
* and some more
* and yet more
* i broke SOMETHING but what, it's a mystery
* tidy up
* vendor ttlcache
* use ttlcache
* fix up some tests
* rename some stuff
* little reminder
* some more updates
* new styling for frontpage, update login and authorize templates
* run go fmt
* add AssetBaseDir to command flag parsing
* untested: move landing page to it's own router
* go fmt, fix typo
* fix package, adapt to proper Route structure
* start messing about with timeline manager
* i have no idea what i'm doing
* i continue to not know what i'm doing
* it's coming along
* bit more progress
* update timeline with new posts as they come in
* lint and fmt
* Select accounts where empty string
* restructure a bunch, get unfaves working
* moving stuff around
* federate status deletes properly
* mention regex better but not 100% there
* fix regex
* some more hacking away at the timeline code phew
* fix up some little things
* i can't even
* more timeline stuff
* move to ulid
* fiddley
* some lil fixes for kibou compatibility
* timelines working pretty alright!
* tidy + lint
Inbox POST from federated servers now working for statuses and follow requests.
Follow request client API added.
Start work on federating outgoing messages.
Other fixes and changes/tidying up.
* start work on accounts module
* plodding away on the accounts endpoint
* groundwork for other account routes
* add password validator
* validation utils
* require account approval flags
* comments
* comments
* go fmt
* comments
* add distributor stub
* rename api to federator
* tidy a bit
* validate new account requests
* rename r router
* comments
* add domain blocks
* add some more shortcuts
* add some more shortcuts
* check email + username availability
* email block checking for signups
* chunking away at it
* tick off a few more things
* some fiddling with tests
* add mock package
* relocate repo
* move mocks around
* set app id on new signups
* initialize oauth server properly
* rename oauth server
* proper mocking tests
* go fmt ./...
* add required fields
* change name of func
* move validation to account.go
* more tests!
* add some file utility tools
* add mediaconfig
* new shortcut
* add some more fields
* add followrequest model
* add notify
* update mastotypes
* mock out storage interface
* start building media interface
* start on update credentials
* mess about with media a bit more
* test image manipulation
* media more or less working
* account update nearly working
* rearranging my package ;) ;) ;)
* phew big stuff!!!!
* fix type checking
* *fiddles*
* Add CreateTables func
* account registration flow working
* tidy
* script to step through auth flow
* add a lil helper for generating user uris
* fiddling with federation a bit
* update progress
* Tidying and linting
* add host and protocol options
* some fiddling
* tidying up and comments
* tick off /oauth/token
* tidying a bit
* tidying
* go mod tidy
* allow attaching middleware to server
* add middleware
* more user friendly
* add comments
* comments
* store account + app
* tidying
* lots of restructuring
* lint + tidy
Dominik Süß