More fix for #24981
* #24981Close#22361
* #22361
There were many patches for Gitea's sub-commands to satisfy the facts:
* Some sub-commands shouldn't output any log, otherwise the git protocol
would be broken
* Sometimes the users want to see "verbose" or "quiet" outputs
That's a longstanding problem, and very fragile. This PR is only a quick
patch for the problem.
In the future, the sub-command system should be refactored to a clear
solution.
----
Other changes:
* Use `ReplaceAllWriters` to replace
`RemoveAllWriters().AddWriters(writer)`, then it's an atomic operation.
* Remove unnecessary `syncLevelInternal` calls, because
`AddWriters/addWritersInternal` already calls it.
Co-authored-by: Giteabot <teabot@gitea.io>
Fix#25481
The `InitWorkPathAndCommonConfig` calls `LoadCommonSettings` which does
many checks like "current user is root or not".
Some commands like "environment-to-ini" shouldn't do such check, because
it might be run with "root" user at the moment (eg: the docker's setup
script)
ps: in the future, the docker's setup script should be improved to avoid
Gitea's command running with "root"
In modern days, there is no reason to make users set "charset" anymore.
Close#25378
## ⚠️ BREAKING
The key `[database].CHARSET` was removed completely as every newer
(>10years) MySQL database supports `utf8mb4` already.
There is a (deliberately) undocumented new fallback option if anyone
still needs to use it, but we don't recommend using it as it simply
causes problems.
# The problem
There were many "path tricks":
* By default, Gitea uses its program directory as its work path
* Gitea tries to use the "work path" to guess its "custom path" and
"custom conf (app.ini)"
* Users might want to use other directories as work path
* The non-default work path should be passed to Gitea by GITEA_WORK_DIR
or "--work-path"
* But some Gitea processes are started without these values
* The "serv" process started by OpenSSH server
* The CLI sub-commands started by site admin
* The paths are guessed by SetCustomPathAndConf again and again
* The default values of "work path / custom path / custom conf" can be
changed when compiling
# The solution
* Use `InitWorkPathAndCommonConfig` to handle these path tricks, and use
test code to cover its behaviors.
* When Gitea's web server runs, write the WORK_PATH to "app.ini", this
value must be the most correct one, because if this value is not right,
users would find that the web UI doesn't work and then they should be
able to fix it.
* Then all other sub-commands can use the WORK_PATH in app.ini to
initialize their paths.
* By the way, when Gitea starts for git protocol, it shouldn't output
any log, otherwise the git protocol gets broken and client blocks
forever.
The "work path" priority is: WORK_PATH in app.ini > cmd arg --work-path
> env var GITEA_WORK_DIR > builtin default
The "app.ini" searching order is: cmd arg --config > cmd arg "work path
/ custom path" > env var "work path / custom path" > builtin default
## ⚠️ BREAKING
If your instance's "work path / custom path / custom conf" doesn't meet
the requirements (eg: work path must be absolute), Gitea will report a
fatal error and exit. You need to set these values according to the
error log.
----
Close#24818Close#24222Close#21606Close#21498Close#25107Close#24981
Maybe close#24503
Replace #23301
Replace #22754
And maybe more
That's a longstanding INI package problem: the "MustXxx" calls change
the option values, and the following "Save" will save a lot of garbage
options into the user's config file.
Ideally we should refactor the INI package to a clear solution, but it's
a huge work.
A clear workaround is what this PR does: when "Save", load a clear INI
instance and save it.
Partially fix#25377, the "install" page needs more fine tunes.
Follow up #22405Fix#20703
This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.
- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The INI package has many bugs and quirks, and in fact it is
unmaintained.
This PR is the first step for the INI package refactoring:
* Use Gitea's "config_provider" to provide INI access
* Deprecate the INI package by golangci.yml rule
It's been disabled by default since 1.17
(https://github.com/go-gitea/gitea/pull/18914), and it never really
delivered any benefit except being another cache layer that has its own
unsolved invalidation issues. HTTP cache works, we don't need two cache
layers at the browser for assets.
## ⚠️ BREAKING
You can remove the config `[ui].USE_SERVICE_WORKER` from your `app.ini`
now.
As discussed in
https://github.com/go-gitea/gitea/pull/24953#issuecomment-1565630156.
## ⚠️ BREAKING ⚠️
1. The `ui.THEME_COLOR_META_TAG` setting has been removed. If you still
need to set the `theme-color` meta tag, add it via
`$GITEA_CUSTOM/templates/custom/header.tmpl` instead.
2. The non-standard `default-theme` meta-tag added in
https://github.com/go-gitea/gitea/pull/13809 has been removed. Third
party code that needs to obtain the currently loaded theme should use
the `theme-<name>` class on the `<html>` node instead, which reflect the
currently active theme.
This adds the ability to pin important Issues and Pull Requests. You can
also move pinned Issues around to change their Position. Resolves#2175.
## Screenshots
The Design was mostly copied from the Projects Board.
## Implementation
This uses a new `pin_order` Column in the `issue` table. If the value is
set to 0, the Issue is not pinned. If it's set to a bigger value, the
value is the Position. 1 means it's the first pinned Issue, 2 means it's
the second one etc. This is dived into Issues and Pull requests for each
Repo.
## TODO
- [x] You can currently pin as many Issues as you want. Maybe we should
add a Limit, which is configurable. GitHub uses 3, but I prefer 6, as
this is better for bigger Projects, but I'm open for suggestions.
- [x] Pin and Unpin events need to be added to the Issue history.
- [x] Tests
- [x] Migration
**The feature itself is currently fully working, so tester who may find
weird edge cases are very welcome!**
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
1. non-dev mode is treated as prod mode, to protect users from
accidentally running in dev mode if there is a typo in this value.
2. in dev mode, do not need to really exit if there are template errors,
because the template errors could be fixed by developer soon and the
templates get reloaded, help:
* https://github.com/go-gitea/gitea/issues/24845#issuecomment-1557615382
3. Fine tune the mail template loading message.
Replace #19857Close#19856Close#10311Close#10123
Major changes:
1. Move a lot of code from `environment-to-ini.go` to `config_env.go` to
make them testable.
2. Add `__FILE` support
3. Update documents
4. Add tests
Close#13454 , Close#23255, Close#14697 (and maybe more related
issues)
Many users have the requirement to customize the git config. This PR
introduces an easy way: put the options in Gitea's app.ini
`[git.config]`, then the config options will be applied to git config.
And it can support more flexible default config values, eg: now
`diff.algorithm=histogram` by default. According to:
https://stackoverflow.com/a/32367597/4754037 , `histogram diff` is
efficient and doesn't like to cause server-side problems.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: Giteabot <teabot@gitea.io>
Replace #20257 (which is stale and incomplete)
Close#20255
Major changes:
* Deprecate the "WHITELIST", use "ALLOWLIST"
* Add wildcard support for EMAIL_DOMAIN_ALLOWLIST/EMAIL_DOMAIN_BLOCKLIST
* Update example config file and document
* Improve tests
## ⚠️ Breaking
The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.
Although many legacy options still work, it's encouraged to upgrade to
the new options.
The SMTP logger is deleted because SMTP is not suitable to collect logs.
If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.
## Description
Close#12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)
Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)
There is a new document (with examples): `logging-config.en-us.md`
This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.
## The old problems
The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.
## The new design
See `logger.go` for documents.
## Screenshot
<details>
</details>
## TODO
* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
Regression of #24536. If the user doesn't explicitly disable Actions, it
will be enabled.
1. Gitea will call `loadRepositoryFrom` before `loadActionsFrom`.
25d4f95df2/modules/setting/setting.go (L234-L237)
2. In `loadRepositoryFrom`,
`rootCfg.Section("actions").Key("ENABLED").MustBool(true)` will set
`actions.ENABLED` with `true`.
25d4f95df2/modules/setting/repository.go (L313-L315)
3. In `loadActionsFrom`, `rootCfg.Section("actions")` will get a section
with Actions enabled.
25d4f95df2/modules/setting/actions.go (L23-L26)
Although the cause of the problem was using `true` by copy-paste
mistake, it also surprised me that
**`rootCfg.Section("actions").Key("ENABLED").MustBool(true)` doesn't
only read, but also write.**
Make mailer SMTP check have timed context
Otherwise Gitea may block for long time if the DNS request blocks.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
fix#12192 Support SSH for go get
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Giteabot <teabot@gitea.io>
Co-authored-by: mfk <mfk@hengwei.com.cn>
Co-authored-by: silverwind <me@silverwind.io>
According to the discussion with DanielGibson, the default "20m" seems
too short. It would make LFS fail if the file is large / network is
slow.
I think relaxing this timeout doesn't have side affect. So change the
default value to 24h, IMO that should be long enough.
## ⚠️ BREAKING
If admins want the previous timeout, they should set the setting
`[server].LFS_HTTP_AUTH_EXPIRY`.
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
close#24449
The unit of `Actions` should be contorlled not only by
`repository.DISABLED_REPO_UNITS` but also by `actions.ENABLED`
in the `app.ini`.
Previously, the permission of the team's `Actions` unit was not
controlled by `actions.Enabled`. So, even if the user sets
`actions.Enabled` to false, he can still select the permission of the
`Actions` unit for the team.
This PR makes the permissions of the team's `Actions` unit also
controlled by `actions.Enabled`. Just append`TypeActions` into
`DisabledRepoUnits` slice when initializing if `actions.Enabled` is
false.
### Changes:
If `Actions` is set disbaled in `app.ini`, like below:
```yaml
[actions]
ENABLED = false
```
1. If user try to create/edit a team, will prompt user that `Actions` is disabled.
2. `actions` is not displayed in the sidebar on the team details page
Co-authored-by: @awkwardbunny
This PR adds a Debian package registry. You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing. Source packages are not supported at the
moment and I did not find documentation of the architecture "all" and
how these packages should be treated.
---------
Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
## Needs and benefits
[Livebook](https://livebook.dev/) notebooks are used for code
documentation and for deep dives and note-taking in the elixir
ecosystem. Rendering these in these as Markdown on frogejo has many
benefits, since livemd is a subset of markdown. Some of the benefits
are:
- New users of elixir and livebook are scared by unformated .livemd
files, but are shown what they expect
- Sharing a notebook is as easy as sharing a link, no need to install
the software in order to see the results.
[goldmark-meraid ](https://github.com/abhinav/goldmark-mermaid) is a
mermaid-js parser already included in gitea. This makes the .livemd
rendering integration feature complete. With this PR class diagrams, ER
Diagrams, flow charts and much more will be rendered perfectly.
With the additional functionality gitea will be an ideal tool for
sharing resources with fellow software engineers working in the elixir
ecosystem. Allowing the git forge to be used without needing to install
any software.
## Feature Description
This issue requests the .livemd extension to be added as a Markdown
language extension.
- `.livemd` is the extension of Livebook which is an Elixir version of
Jupyter Notebook.
- `.livemd` is` a subset of Markdown.
This would require the .livemd to be recognized as a markdown file. The
Goldmark the markdown parser should handle the parsing and rendering
automatically.
Here is the corresponding commit for GitHub linguist:
https://github.com/github/linguist/pull/5672
Here is a sample page of a livemd file:
https://github.com/github/linguist/blob/master/samples/Markdown/livebook.livemd
## Screenshots
The first screenshot shows how github shows the sample .livemd in the
browser.
The second screenshot shows how mermaid js, renders my development
notebook and its corresponding ER Diagram. The source code can be found
here:
79615f7428/termiNotes.livemd
## Testing
I just changed the file extension from `.livemd`to `.md`and the document
already renders perfectly on codeberg. Check you can it out
[here](https://codeberg.org/lgh/Termi/src/branch/livemd2md/termiNotes.md)
---------
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
The `SHOW_FOOTER_BRANDING` came from year 2015, and it seems nobody ever
uses it. It only shows an GitHub icon which seems unrelated to Gitea, it
doesn't do what document says. So, remove it.
## ⚠️ Breaking
Users can now remove the key `[other].SHOW_FOOTER_BRANDING` from their
app.ini.
Close#24062
At the beginning, I just wanted to fix the warning mentioned by #24062
But, the cookie code really doesn't look good to me, so clean up them.
Complete the TODO on `SetCookie`:
> TODO: Copied from gitea.com/macaron/macaron and should be improved
after macaron removed.
The default access log format has been unnecessarily escaped, leading to
spurious backslashes appearing in log lines.
Additionally, the `RemoteAddr` field includes the port, which breaks
most log parsers attempting to process it. I've added a call to
`net.SplitHostPort()` attempting to isolate the address alone, with a
fallback to the original address if it errs.
Signed-off-by: Gary Moon <gary@garymoon.net>
The idea is to use a Layered Asset File-system (modules/assetfs/layered.go)
For example: when there are 2 layers: "custom", "builtin", when access
to asset "my/page.tmpl", the Layered Asset File-system will first try to
use "custom" assets, if not found, then use "builtin" assets.
This approach will hugely simplify a lot of code, make them testable.
Other changes:
* Simplify the AssetsHandlerFunc code
* Simplify the `gitea embedded` sub-command code
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Without this patch, the setting SSH.StartBuiltinServer decides whether
the native (Go) implementation is used rather than calling 'ssh-keygen'.
It's possible for 'using ssh-keygen' and 'using the built-in server' to
be independent.
In fact, the gitea rootless container doesn't ship ssh-keygen and can be
configured to use the host's SSH server - which will cause the public
key parsing mechanism to break.
This commit changes the decision to be based on SSH.KeygenPath instead.
Any existing configurations with a custom KeygenPath set will continue
to function. The new default value of '' selects the native version. The
downside of this approach is that anyone who has relying on plain
'ssh-keygen' to have special properties will now be using the native
version instead.
I assume the exec-variant is only there because /x/crypto/ssh didn't
support ssh-ed25519 until 2016. I don't see any other reason for using
it so it might be an acceptable risk.
Fixes#23363
EDIT: this message was garbled when I tried to get the commit
description back in.. Trying to reconstruct it:
## ⚠️ BREAKING ⚠️ Users who don't have SSH.KeygenPath
explicitly set and rely on the ssh-keygen binary need to set
SSH.KeygenPath to 'ssh-keygen' in order to be able to continue using it
for public key parsing.
There was something else but I can't remember at the moment.
EDIT2: It was about `make test` and `make lint`. Can't get them to run.
To reproduce the issue, I installed `golang` in `docker.io/node:16` and
got:
```
...
go: mvdan.cc/xurls/v2@v2.4.0: unknown revision mvdan.cc/xurls/v2.4.0
go: gotest.tools/v3@v3.4.0: unknown revision gotest.tools/v3.4.0
...
go: gotest.tools/v3@v3.0.3: unknown revision gotest.tools/v3.0.3
...
go: error loading module requirements
```
Signed-off-by: Leon M. Busch-George <leon@georgemail.eu>
Follow #21962
After I eat my own dogfood, I would say that
ONLY_SHOW_RELEVANT_REPOS=false is necessary for many private/enterprise
instances, because many private repositories do not have
"description/topic", users just want to search by their names.
This PR also adds `PageIsExploreRepositories` check, to make code more
strict, because the `search` template is shared for different purpose.
And during the test, I found a bug that the "Search" button didn't
respect the "relevant" parameter, so this PR fixes the bug by the way
together.
I think this PR needs to be backported.
Remove `[repository.editor] PREVIEWABLE_FILE_MODES` setting that seemed
like it was intended to support this but did not work. Instead, whenever
viewing a file shows a preview, also have a Preview tab in the file
editor.
Add new `/markup` web and API endpoints with `comment`, `gfm`,
`markdown` and new `file` mode that uses a file path to determine the
renderer.
Remove `/markdown` web endpoint but keep the API for backwards and
GitHub compatibility.
## ⚠️ BREAKING ⚠️
The `[repository.editor] PREVIEWABLE_FILE_MODES` setting was removed.
This setting served no practical purpose and was not working correctly.
Instead a preview tab is always shown in the file editor when supported.
---------
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Creating pid file should not belong to setting package and only web
command needs that. So this PR moves pidfile creation from setting
package to web command package to keep setting package more readable.
I marked this as `break` because the PIDFile path moved. For those who
have used the pid build argument, it has to be changed.
---------
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
This PR adds support for reflogs on all repositories. It does this by
adding a global configuration entry.
Implements #14865
---------
Signed-off-by: Philip Peterson <philip.c.peterson@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Replace #23350.
Refactor `setting.Database.UseMySQL` to
`setting.Database.Type.IsMySQL()`.
To avoid mismatching between `Type` and `UseXXX`.
This refactor can fix the bug mentioned in #23350, so it should be
backported.
During the recent hash algorithm change it became clear that the choice
of password hash algorithm plays a role in the time taken for CI to run.
Therefore as attempt to improve CI we should consider using a dummy
hashing algorithm instead of a real hashing algorithm.
This PR creates a dummy algorithm which is then set as the default
hashing algorithm during tests that use the fixtures. This hopefully
will cause a reduction in the time it takes for CI to run.
---------
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Some bugs caused by less unit tests in fundamental packages. This PR
refactor `setting` package so that create a unit test will be easier
than before.
- All `LoadFromXXX` files has been splited as two functions, one is
`InitProviderFromXXX` and `LoadCommonSettings`. The first functions will
only include the code to create or new a ini file. The second function
will load common settings.
- It also renames all functions in setting from `newXXXService` to
`loadXXXSetting` or `loadXXXFrom` to make the function name less
confusing.
- Move `XORMLog` to `SQLLog` because it's a better name for that.
Maybe we should finally move these `loadXXXSetting` into the `XXXInit`
function? Any idea?
---------
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
This PR refactors and improves the password hashing code within gitea
and makes it possible for server administrators to set the password
hashing parameters
In addition it takes the opportunity to adjust the settings for `pbkdf2`
in order to make the hashing a little stronger.
The majority of this work was inspired by PR #14751 and I would like to
thank @boppy for their work on this.
Thanks to @gusted for the suggestion to adjust the `pbkdf2` hashing
parameters.
Close#14751
---------
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Added a new captcha(cloudflare turnstile) and its corresponding
document. Cloudflare turnstile official instructions are here:
https://developers.cloudflare.com/turnstile
Signed-off-by: ByLCY <bylcy@bylcy.dev>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Jason Song <i@wolfogre.com>
Every user can already disable the filter manually, so the explicit
setting is absolutely useless and only complicates the logic.
Previously, there was also unexpected behavior when multiple query
parameters were present.
---------
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Most of the time forks are used for contributing code only, so not
having
issues, projects, release and packages is a better default for such
cases.
They can still be enabled in the settings.
A new option `DEFAULT_FORK_REPO_UNITS` is added to configure the default
units on forks.
Also add missing `repo.packages` unit to documentation.
code by: @brechtvl
## ⚠️ BREAKING ⚠️
When forking a repository, the fork will now have issues, projects,
releases, packages and wiki disabled. These can be enabled in the
repository settings afterwards. To change back to the previous default
behavior, configure `DEFAULT_FORK_REPO_UNITS` to be the same value as
`DEFAULT_REPO_UNITS`.
Co-authored-by: Brecht Van Lommel <brecht@blender.org>
closes#13585fixes#9067fixes#2386
ref #6226
ref #6219fixes#745
This PR adds support to process incoming emails to perform actions.
Currently I added handling of replies and unsubscribing from
issues/pulls. In contrast to #13585 the IMAP IDLE command is used
instead of polling which results (in my opinion 😉) in cleaner code.
Procedure:
- When sending an issue/pull reply email, a token is generated which is
present in the Reply-To and References header.
- IMAP IDLE waits until a new email arrives
- The token tells which action should be performed
A possible signature and/or reply gets stripped from the content.
I added a new service to the drone pipeline to test the receiving of
incoming mails. If we keep this in, we may test our outgoing emails too
in future.
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
#18058 made a mistake. The disableGravatar's default value depends on
`OfflineMode`. If it's `true`, then `disableGravatar` is true, otherwise
it's `false`. But not opposite.
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
If user has reached the maximum limit of repositories:
- Before
- disallow create
- allow fork without limit
- This patch:
- disallow create
- disallow fork
- Add option `ALLOW_FORK_WITHOUT_MAXIMUM_LIMIT` (Default **true**) :
enable this allow user fork repositories without maximum number limit
fixed https://github.com/go-gitea/gitea/issues/21847
Signed-off-by: Xinyu Zhou <i@sourcehut.net>
Since #20218 introduced internal watching template, template watching
should be removed from `air`. This will prevent restart the whole server
once the template files changed to speed up developing when using `make
watch`.
To ensure `make watch` will reuse template watching, this PR introduced
a new ENV `GITEA_RUN_MODE` to make sure `make watch` will always run in
a dev mode of Gitea so that template watching will open.
This PR also added more exclude testdata directories.
For a long time Gitea has tested PR patches using a git apply --check
method, and in fact prior to the introduction of a read-tree assisted
three-way merge in #18004, this was the only way of checking patches.
Since #18004, the git apply --check method has been a fallback method,
only used when the read-tree three-way merge method has detected a
conflict. The read-tree assisted three-way merge method is much faster
and less resource intensive method of detecting conflicts. #18004 kept
the git apply method around because it was thought possible that this
fallback might be able to rectify conflicts that the read-tree three-way
merge detected. I am not certain if this could ever be the case.
Given the uncertainty here and the now relative stability of the
read-tree method - this PR makes using this fallback optional and
disables it by default. The hope is that users will not notice any
significant difference in conflict detection and we will be able to
remove the git apply fallback in future, and/or improve the read-tree
three-way merge method to catch any conflicts that git apply method
might have been able to fix.
An additional benefit is that patch checking should be significantly
less resource intensive and much quicker.
(See
https://github.com/go-gitea/gitea/issues/22083\#issuecomment-1347961737)
Ref #22083
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Gitea will attempt to lookup its location using LookPath however, this
fails on cmd.exe if gitea is in the current working directory.
exec.LookPath will return an exec.ErrDot error which we can test for and
then simply using filepath.Abs(os.Args[0]) to absolute gitea against the
current working directory.
Fix#22063
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Gitea will migrate the database model version automatically, but it
should be able to be disabled and keep Gitea shutdown if the version is
not matched.
Fixes#21865.
Scheme-based normalization ([RFC 3986, section
6.2.3](https://www.rfc-editor.org/rfc/rfc3986#section-6.2.3)) was
already implemented, but only for `defaultAppURL`.
This PR implements the same for `AppURL`.
Signed-off-by: Saswat Padhi <saswatpadhi@protonmail.com>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Change all license headers to comply with REUSE specification.
Fix#16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Unfortunately the fallback configuration code for [mailer] that were
added in #18982 are incorrect. When you read a value from an ini section
that key is added. This leads to a failure of the fallback mechanism.
Further there is also a spelling mistake in the startTLS configuration.
This PR restructures the mailer code to first map the deprecated
settings on to the new ones - and then use ini.MapTo to map those on to
the struct with additional validation as necessary.
Ref #21744
Signed-off-by: Andrew Thornton <art27@cantab.net>
Enable this to require captcha validation for user login. You also must
enable `ENABLE_CAPTCHA`.
Summary:
- Consolidate CAPTCHA template
- add CAPTCHA handle and context
- add `REQUIRE_CAPTCHA_FOR_LOGIN` config and docs
- Consolidate CAPTCHA set-up and verification code
Partially resolved#6049
Signed-off-by: Xinyu Zhou <i@sourcehut.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
This patch provide a mechanism to disable RSS/Atom feed.
Signed-off-by: Xinyu Zhou <i@sourcehut.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
This PR enhances the CORS middleware usage by allowing for the headers
to be configured in `app.ini`.
Fixes#21746
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Attempt clarify the AppWorkPath in the documentation by using different
notation and adding a section to the start of the cheat sheet.
Fix#21523
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Related #20471
This PR adds global quota limits for the package registry. Settings for
individual users/orgs can be added in a seperate PR using the settings
table.
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Fix https://github.com/go-gitea/gitea/issues/21666
Caused by https://github.com/go-gitea/gitea/pull/19663
Before: when install, the INTERNAL_TOKEN was always generated and saved.
But the internal token may be already there by INTERNAL_TOKEN_URI
After: INTERNAL_TOKEN_URI file must be non-empty. When install, skip
internal token generation if the token exists.
This patch provide a mechanism to disable
[sitemap](https://github.com/go-gitea/gitea/pull/18407).
Signed-off-by: Xinyu Zhou <i@sourcehut.net>
Co-authored-by: 6543 <6543@obermui.de>
Only load SECRET_KEY and INTERNAL_TOKEN if they exist.
Never write the config file if the keys do not exist, which was only a fallback for Gitea upgraded from < 1.5
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This PR adds mathematical rendering with KaTeX.
The first step is to add a Goldmark extension that detects the latex
(and tex) mathematics delimiters.
The second step to make this extension only run if math support is
enabled.
The second step is to then add KaTeX CSS and JS to the head which will
load after the dom is rendered.
Fix#3445
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Adds a new option to only show relevant repo's on the explore page, for bigger Gitea instances like Codeberg this is a nice option to enable to make the explore page more populated with unique and "high" quality repo's. A note is shown that the results are filtered and have the possibility to see the unfiltered results.
Co-authored-by: vednoc <vednoc@protonmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
This should solve the main problem of dynamic assets getting stale after
a version upgrade. Everything not affected will use query-string based
cache busting, which includes files loaded via HTML or worker scripts.
This PR adds functionality to allow Gitea to sit behind an
HAProxy and HAProxy protocolled connections directly.
Fix#7508
Signed-off-by: Andrew Thornton <art27@cantab.net>
This adds support for getting the user's full name from the reverse
proxy in addition to username and email.
Tested locally with caddy serving as reverse proxy with Tailscale
authentication.
Signed-off-by: Will Norris <will@tailscale.com>
Signed-off-by: Will Norris <will@tailscale.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* increase default page sizes to 20
* adjust docs to reflect that the setting is used all over the place
* fix tests
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* `PROTOCOL`: can be smtp, smtps, smtp+startls, smtp+unix, sendmail, dummy
* `SMTP_ADDR`: domain for SMTP, or path to unix socket
* `SMTP_PORT`: port for SMTP; defaults to 25 for `smtp`, 465 for `smtps`, and 587 for `smtp+startls`
* `ENABLE_HELO`, `HELO_HOSTNAME`: reverse `DISABLE_HELO` to `ENABLE_HELO`; default to false + system hostname
* `FORCE_TRUST_SERVER_CERT`: replace the unclear `SKIP_VERIFY`
* `CLIENT_CERT_FILE`, `CLIENT_KEY_FILE`, `USE_CLIENT_CERT`: clarify client certificates here
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Use default values when provided values are empty
- When provided values are empty like `:3000` would imply that host is
empty, use the default value.
- Resolves#20316
* Update database.go
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Creating the directory automatically is not correct.
In other places for ssh key writing (RewriteAllPrincipalKeys / appendAuthorizedKeysToFile, etc), the directory will still be created when updating the keys.
This PR will resolve the confusing and annoying problem: the dummy and empty ".ssh" directory in new git home.
* Add git.HOME_PATH
* add legacy file check
* Apply suggestions from code review
Co-authored-by: zeripath <art27@cantab.net>
* pass env GNUPGHOME to git command, move the existing .gitconfig to new home, make the fix for 1.17rc more clear.
* set git.HOME_PATH for docker images to default HOME
* Revert "set git.HOME_PATH for docker images to default HOME"
This reverts commit f120101ddc.
* force Gitea to use a stable GNUPGHOME directory
* extra check to ensure only process dir or symlink for legacy files
* refactor variable name
* The legacy dir check (for 1.17-rc1) could be removed with 1.18 release, since users should have upgraded from 1.17-rc to 1.17-stable
* Update modules/git/git.go
Co-authored-by: Steven Kriegler <61625851+justusbunsi@users.noreply.github.com>
* remove initFixGitHome117rc
* Update git.go
* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Steven Kriegler <61625851+justusbunsi@users.noreply.github.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Unfortunately it appears that 2048 bit RSA keys can occasionally be created in such
a way that they appear to have 2047 bit length. This PR simply changes our defaults to
allow these.
Fix#20249
Signed-off-by: Andrew Thornton <art27@cantab.net>
- Completely remove U2F support from 1.18.0, 1.17.0 will be the last
release that U2F is somewhat supported. Users who used U2F would already
be warned about using U2F for a while now and should hopefully already
be migrated. But starting 1.18 definitely remove it.
* go.mod: add go-fed/{httpsig,activity/pub,activity/streams} dependency
go get github.com/go-fed/activity/streams@master
go get github.com/go-fed/activity/pub@master
go get github.com/go-fed/httpsig@master
* activitypub: implement /api/v1/activitypub/user/{username} (#14186)
Return informations regarding a Person (as defined in ActivityStreams
https://www.w3.org/TR/activitystreams-vocabulary/#dfn-person).
Refs: https://github.com/go-gitea/gitea/issues/14186
Signed-off-by: Loïc Dachary <loic@dachary.org>
* activitypub: add the public key to Person (#14186)
Refs: https://github.com/go-gitea/gitea/issues/14186
Signed-off-by: Loïc Dachary <loic@dachary.org>
* activitypub: go-fed conformant Clock instance
Signed-off-by: Loïc Dachary <loic@dachary.org>
* activitypub: signing http client
Signed-off-by: Loïc Dachary <loic@dachary.org>
* activitypub: implement the ReqSignature middleware
Signed-off-by: Loïc Dachary <loic@dachary.org>
* activitypub: hack_16834
Signed-off-by: Loïc Dachary <loic@dachary.org>
* Fix CI checks-backend errors with go mod tidy
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Change 2021 to 2022, properly format package imports
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Run make fmt and make generate-swagger
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Use Gitea JSON library, add assert for pkp
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Run make fmt again, fix err var redeclaration
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Remove LogSQL from ActivityPub person test
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Assert if json.Unmarshal succeeds
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Cleanup, handle invalid usernames for ActivityPub person GET request
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Rename hack_16834 to user_settings
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Use the httplib module instead of http for GET requests
* Clean up whitespace with make fmt
* Use time.RFC1123 and make the http.Client proxy-aware
* Check if digest algo is supported in setting module
* Clean up some variable declarations
* Remove unneeded copy
* Use system timezone instead of setting.DefaultUILocation
* Use named constant for httpsigExpirationTime
* Make pubKey IRI #main-key instead of /#main-key
* Move /#main-key to #main-key in tests
* Implemented Webfinger endpoint.
* Add visible check.
* Add user profile as alias.
* Add actor IRI and remote interaction URL to WebFinger response
* fmt
* Fix lint errors
* Use go-ap instead of go-fed
* Run go mod tidy to fix missing modules in go.mod and go.sum
* make fmt
* Convert remaining code to go-ap
* Clean up go.sum
* Fix JSON unmarshall error
* Fix CI errors by adding @context to Person() and making sure types match
* Correctly decode JSON in api_activitypub_person_test.go
* Force CI rerun
* Fix TestActivityPubPersonInbox segfault
* Fix lint error
* Use @mariusor's suggestions for idiomatic go-ap usage
* Correctly add inbox/outbox IRIs to person
* Code cleanup
* Remove another LogSQL from ActivityPub person test
* Move httpsig algos slice to an init() function
* Add actor IRI and remote interaction URL to WebFinger response
* Update TestWebFinger to check for ActivityPub IRI in aliases
* make fmt
* Force CI rerun
* WebFinger: Add CORS header and fix Href -> Template for remote interactions
The CORS header is needed due to https://datatracker.ietf.org/doc/html/rfc7033#section-5 and fixes some Peertube <-> Gitea federation issues
* make lint-backend
* Make sure Person endpoint has Content-Type application/activity+json and includes PreferredUsername, URL, and Icon
Setting the correct Content-Type is essential for federating with Mastodon
* Use UTC instead of GMT
* Rename pkey to pubKey
* Make sure HTTP request Date in GMT
* make fmt
* dont drop err
* Make sure API responses always refer to username in original case
Copied from what I wrote on #19133 discussion: Handling username case is a very tricky issue and I've already encountered a Mastodon <-> Gitea federation bug due to Gitea considering Ta180m and ta180m to be the same user while Mastodon thinks they are two different users. I think the best way forward is for Gitea to only use the original case version of the username for federation so other AP software don't get confused.
* Move httpsig algs constant slice to modules/setting/federation.go
* Add new federation settings to app.example.ini and config-cheat-sheet
* Return if marshalling error
* Make sure Person IRIs are generated correctly
This commit ensures that if the setting.AppURL is something like "http://127.0.0.1:42567" (like in the integration tests), a trailing slash will be added after that URL.
* If httpsig verification fails, fix Host header and try again
This fixes a very rare bug when Gitea and another AP server (confirmed to happen with Mastodon) are running on the same machine, Gitea fails to verify incoming HTTP signatures. This is because the other AP server creates the sig with the public Gitea domain as the Host. However, when Gitea receives the request, the Host header is instead localhost, so the signature verification fails. Manually changing the host header to the correct value and trying the veification again fixes the bug.
* Revert "If httpsig verification fails, fix Host header and try again"
This reverts commit f53e46c721.
The bug was actually caused by nginx messing up the Host header when reverse-proxying since I didn't have the line `proxy_set_header Host $host;` in my nginx config for Gitea.
* Go back to using ap.IRI to generate inbox and outbox IRIs
* use const for key values
* Update routers/web/webfinger.go
* Use ctx.JSON in Person response to make code cleaner
* Revert "Use ctx.JSON in Person response to make code cleaner"
This doesn't work because the ctx.JSON() function already sends the response out and it's too late to edit the headers.
This reverts commit 95aad98897.
* Use activitypub.ActivityStreamsContentType for Person response Content Type
* Limit maximum ActivityPub request and response sizes to a configurable setting
* Move setting key constants to models/user/setting_keys.go
* Fix failing ActivityPubPerson integration test by checking the correct field for username
* Add a warning about changing settings that can break federation
* Add better comments
* Don't multiply Federation.MaxSize by 1<<20 twice
* Add more better comments
* Fix failing ActivityPubMissingPerson test
We now use ctx.ContextUser so the message printed out when a user does not exist is slightly different
* make generate-swagger
For some reason I didn't realize that /templates/swagger/v1_json.tmpl was machine-generated by make generate-swagger... I've been editing it by hand for three months! 🤦
* Move getting the RFC 2616 time to a separate function
* More code cleanup
* Update go-ap to fix empty liked collection and removed unneeded HTTP headers
* go mod tidy
* Add ed25519 to httpsig algorithms
* Use go-ap/jsonld to add @context and marshal JSON
* Change Gitea user agent from the default to Gitea/Version
* Use ctx.ServerError and remove all remote interaction code from webfinger.go
* Allow render HTML with css/js external links
* Fix bug because of filename escape chars
* Fix lint
* Update docs about new configuration item
* Fix bug of render HTML in sub directory
* Add CSP head for displaying iframe in rendering file
* Fix test
* Apply suggestions from code review
Co-authored-by: delvh <dev.lh@web.de>
* Some improvements
* some improvement
* revert change in SanitizerDisabled of external renderer
* Add sandbox for iframe and support allow-scripts and allow-same-origin
* refactor
* fix
* fix lint
* fine tune
* use single option RENDER_CONTENT_MODE, use sandbox=allow-scripts
* fine tune CSP
* Apply suggestions from code review
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Move access and repo permission to models/perm/access
* fix test
* Move some git related files into sub package models/git
* Fix build
* fix git test
* move lfs to sub package
* move more git related functions to models/git
* Move functions sequence
* Some improvements per @KN4CK3R and @delvh
* Make AppDataPath absolute against the AppWorkPath if it is not
There are multiple repeated issues whereby a non-absolute provided
APP_DATA_PATH causes strange issues.
This PR simply absolutes the APP_DATA_PATH against the AppWorkPath if
its not so. It also ensures that AppWorkPath is also always absolute.
Ref #19367
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add logging
Signed-off-by: Andrew Thornton <art27@cantab.net>
* absolute workpath against pwd instead of app path first
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Update go tool dependencies
Updated all tool dependencies to latest tags, hoping CI will like it.
* fix new lint errors
* handle more strings.Title cases
* remove lint skip
The main purpose is to refactor the legacy `unknwon/com` package.
1. Remove most imports of `unknwon/com`, only `util/legacy.go` imports the legacy `unknwon/com`
2. Use golangci's depguard to process denied packages
3. Fix some incorrect values in golangci.yml, eg, the version should be quoted string `"1.18"`
4. Use correctly escaped content for `go-import` and `go-source` meta tags
5. Refactor `com.Expand` to our stable (and the same fast) `vars.Expand`, our `vars.Expand` can still return partially rendered content even if the template is not good (eg: key mistach).
The service worker causes a lot of issues with JS errors after instance
upgrades while not bringing any real performance gain over regular HTTP
caching.
Disable it by default for this reason. Maybe later we can remove it
completely, as I simply see no benefit in having it.
- Use a better and more curated list of Ciphers and KeyExchanges, these roughly follows OpenSSH's default.
- Remove some cryptography values which were deprecated.
WebAuthn may cause a security exception if the provided APP_ID is not allowed for the
current origin. Therefore we should reattempt authentication without the appid
extension.
Also we should allow [u2f] as-well as [U2F] sections.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Several users run Gitea in situations whereby `bash` is not available.
If the `SCRIPT_TYPE` is not changed this will cause hooks to fail.
A simple test to check if the provided type is on the PATH should be
sufficient to warn them about this problem.
Signed-off-by: Andrew Thornton <art27@cantab.net>
wxiaoguang
