Frédéric Guillot
8c3f280f32
test(readability): add test case for ExtractContent with broken reader
2025-07-01 20:14:52 -07:00
jvoisin
8a98926674
refactor(readability): add a getSelectionLength function
...
When we're only interested in the length of contained Text, there is no need to
materialize it fully to then call len() on the result: we can simply iterate
over the text element and sum their length instead.
2025-07-01 19:52:53 -07:00
jvoisin
435a950d64
refactor(sanitizer): minor refactorization
...
Use a proper switch-case instead of a bunch of if.
2025-07-01 19:48:55 -07:00
jvoisin
89c32d518d
perf(readability): significantly improve transformMisusedDivsIntoParagraphs
2025-07-01 19:44:58 -07:00
jvoisin
2f7b2e7375
perf(readability): improve getLinkDensity
...
- There is no need to materialize all the content of a given Node when we can
simply compute its length directly, saving a lot of memory, on the order of
several megabytes on my instance, with peaks at a couple of dozen.
- One might object to the usage of a recursive construct, but this is a direct
port of goquery's Text method, so this change doesn't make anything worse.
- The computation of linkLength can be similarly computed, but this can go in
another commit, as it's a bit trickier, since we need to get the length of
every Node that has a `a` Node as parent, without iterating on the whole
parent chain every time.
2025-07-01 19:40:47 -07:00
Frédéric Guillot
6eeccae7cd
test(readability): increase test coverage
2025-06-30 21:29:07 -07:00
jvoisin
99c5bcdb01
perf(storage): truncate strings on go's side instead of pgsql's
...
There is no need to send the whole title and content to have them truncated on
postgresql's side when we can do this client-side. This should save some
memory on the database's side, as well as some bandwidth
if it's located on another server. And it makes the SQL queries a tad more
readable as well.
2025-06-30 19:45:45 -07:00
jvoisin
aed99e65c1
perf(readability): improve getClassWeight speed
...
Before
```console
$ go test -bench=.
goos: linux
goarch: arm64
pkg: miniflux.app/v2/internal/reader/readability
BenchmarkExtractContent-8 34 86102474 ns/op
BenchmarkGetWeight-8 10573 103045 ns/op
PASS
ok miniflux.app/v2/internal/reader/readability 5.409s
```
After
```console
$ go test -bench=.
goos: linux
goarch: arm64
pkg: miniflux.app/v2/internal/reader/readability
BenchmarkExtractContent-8 56 83130924 ns/op
BenchmarkGetWeight-8 246541 5241 ns/op
PASS
ok miniflux.app/v2/internal/reader/readability 6.026s
```
This should make ProcessFeedEntries marginally faster, while saving
some memory.
2025-06-30 19:28:20 -07:00
jvoisin
d1a3f98df9
perf(fetcher): save 8 bytes in the RequestBuilder struct
...
before:
```
// request_builder.go:25 | Size: 64 (Optimal: 56)
type RequestBuilder struct {
headers http.Header ■ ■ ■ ■ ■ ■ ■ ■
clientProxyURL *url.URL ■ ■ ■ ■ ■ ■ ■ ■
useClientProxy bool ■ □ □ □ □ □ □ □
clientTimeout int ■ ■ ■ ■ ■ ■ ■ ■
withoutRedirects bool ■
ignoreTLSErrors bool ■
disableHTTP2 bool ■ □ □ □ □ □
proxyRotator *proxyrotator.ProxyRotator ■ ■ ■ ■ ■ ■ ■ ■
feedProxyURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
}
```
after:
```
// request_builder.go:25 | Size: 56
type RequestBuilder struct {
headers http.Header ■ ■ ■ ■ ■ ■ ■ ■
clientProxyURL *url.URL ■ ■ ■ ■ ■ ■ ■ ■
clientTimeout int ■ ■ ■ ■ ■ ■ ■ ■
useClientProxy bool ■
withoutRedirects bool ■
ignoreTLSErrors bool ■
disableHTTP2 bool ■ □ □ □ □
proxyRotator *proxyrotator.ProxyRotator ■ ■ ■ ■ ■ ■ ■ ■
feedProxyURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
}
```
2025-06-29 16:10:35 -07:00
jvoisin
112494bb66
perf(feed): save 16 bytes in the Feed struct
...
before:
```
// feed.go:25 | Size: 560 (Optimal: 544)
type Feed struct {
ID int64 ■ ■ ■ ■ ■ ■ ■ ■
UserID int64 ■ ■ ■ ■ ■ ■ ■ ■
FeedURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
SiteURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Title string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Description string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
CheckedAt time.Time ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
NextCheckAt time.Time ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
EtagHeader string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
LastModifiedHeader string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
ParsingErrorMsg string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
ParsingErrorCount int ■ ■ ■ ■ ■ ■ ■ ■
ScraperRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
RewriteRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Crawler bool ■ □ □ □ □ □ □ □
BlocklistRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
KeeplistRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
BlockFilterEntryRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
KeepFilterEntryRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
UrlRewriteRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
UserAgent string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Cookie string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Username string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Password string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Disabled bool ■
NoMediaPlayer bool ■
IgnoreHTTPCache bool ■
AllowSelfSignedCertificates bool ■
FetchViaProxy bool ■
HideGlobally bool ■
DisableHTTP2 bool ■ □
AppriseServiceURLs string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
WebhookURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
NtfyEnabled bool ■ □ □ □ □ □ □ □
NtfyPriority int ■ ■ ■ ■ ■ ■ ■ ■
NtfyTopic string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
PushoverEnabled bool ■ □ □ □ □ □ □ □
PushoverPriority int ■ ■ ■ ■ ■ ■ ■ ■
ProxyURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Category *Category ■ ■ ■ ■ ■ ■ ■ ■
Icon *FeedIcon ■ ■ ■ ■ ■ ■ ■ ■
Entries Entries ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
TTL int ■ ■ ■ ■ ■ ■ ■ ■
IconURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
UnreadCount int ■ ■ ■ ■ ■ ■ ■ ■
ReadCount int ■ ■ ■ ■ ■ ■ ■ ■
NumberOfVisibleEntries int ■ ■ ■ ■ ■ ■ ■ ■
}
```
after:
```
// feed.go:25 | Size: 544
type Feed struct {
ID int64 ■ ■ ■ ■ ■ ■ ■ ■
UserID int64 ■ ■ ■ ■ ■ ■ ■ ■
FeedURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
SiteURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Title string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Description string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
CheckedAt time.Time ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
NextCheckAt time.Time ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
EtagHeader string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
LastModifiedHeader string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
ParsingErrorMsg string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
ParsingErrorCount int ■ ■ ■ ■ ■ ■ ■ ■
ScraperRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
RewriteRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
BlocklistRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
KeeplistRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
BlockFilterEntryRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
KeepFilterEntryRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
UrlRewriteRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
UserAgent string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Cookie string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Username string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Password string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Disabled bool ■
NoMediaPlayer bool ■
IgnoreHTTPCache bool ■
AllowSelfSignedCertificates bool ■
FetchViaProxy bool ■
HideGlobally bool ■
DisableHTTP2 bool ■
PushoverEnabled bool ■
NtfyEnabled bool ■
Crawler bool ■ □ □ □ □ □ □
AppriseServiceURLs string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
WebhookURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
NtfyPriority int ■ ■ ■ ■ ■ ■ ■ ■
NtfyTopic string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
PushoverPriority int ■ ■ ■ ■ ■ ■ ■ ■
ProxyURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Category *Category ■ ■ ■ ■ ■ ■ ■ ■
Icon *FeedIcon ■ ■ ■ ■ ■ ■ ■ ■
Entries Entries ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
TTL int ■ ■ ■ ■ ■ ■ ■ ■
IconURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
UnreadCount int ■ ■ ■ ■ ■ ■ ■ ■
ReadCount int ■ ■ ■ ■ ■ ■ ■ ■
NumberOfVisibleEntries int ■ ■ ■ ■ ■ ■ ■ ■
}
```
2025-06-29 16:10:35 -07:00
jvoisin
9f7ecdb75a
perf(model): save 16 bytes in the FeedCreationRequest struct
...
before:
```
// feed.go:154 | Size: 240 (Optimal: 224)
type FeedCreationRequest struct {
FeedURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
CategoryID int64 ■ ■ ■ ■ ■ ■ ■ ■
UserAgent string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Cookie string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Username string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Password string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Crawler bool ■
Disabled bool ■
NoMediaPlayer bool ■
IgnoreHTTPCache bool ■
AllowSelfSignedCertificates bool ■
FetchViaProxy bool ■ □ □
UrlRewriteRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
DisableHTTP2 bool ■ □ □ □ □ □ □ □
ScraperRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
RewriteRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
BlocklistRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
KeeplistRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
BlockFilterEntryRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
KeepFilterEntryRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
HideGlobally bool ■ □ □ □ □ □ □ □
ProxyURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
}
```
after:
```
// feed.go:154 | Size: 224
type FeedCreationRequest struct {
FeedURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
CategoryID int64 ■ ■ ■ ■ ■ ■ ■ ■
UserAgent string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Cookie string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Username string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Password string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
Crawler bool ■
Disabled bool ■
NoMediaPlayer bool ■
IgnoreHTTPCache bool ■
AllowSelfSignedCertificates bool ■
FetchViaProxy bool ■
HideGlobally bool ■
DisableHTTP2 bool ■
ScraperRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
RewriteRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
BlocklistRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
KeeplistRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
BlockFilterEntryRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
KeepFilterEntryRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
UrlRewriteRules string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
ProxyURL string ■ ■ ■ ■ ■ ■ ■ ■
■ ■ ■ ■ ■ ■ ■ ■
}
```
2025-06-29 16:10:35 -07:00
jvoisin
4e1f836266
refactor(readability): simplify a bit getArticle
...
- Use a proper division instead of multiplying by a float.
- Extract a condition in the parent scope
- Use an else-if construct instead of a simple if
2025-06-29 16:06:34 -07:00
Frédéric Guillot
a68de4ee6a
test(readability): add tests for getArticle function
2025-06-29 16:03:17 -07:00
jvoisin
c064891314
perf(readability): Simplify removeUnlikelyCandidates
...
- Use an array of strings instead of a regex, like done in ef13756b1a7a7ba30fd34174a5367381fd8b4849
- Extract the `shouldRemove` function from `removeUnlikelyCandidates`, as there
is no reason to have it there instead of being a proper standalone function.
- Improve a condition, where the goquery selection would have its `id`
attribute left unchecked if a `class` one was present, regardless of if
`class` was a candidate to removal or not.
- Add some comments
2025-06-29 15:31:01 -07:00
Frédéric Guillot
5129f53d58
test(readability): add tests for removeUnlikelyCandidates function
2025-06-29 15:23:56 -07:00
Frédéric Guillot
e60f0fd142
test(readability): add tests for getClassWeight function
2025-06-29 13:24:06 -07:00
Julien Voisin
2b26a345cd
perf(processor): minify content even further
...
There is no need to keep comments (conditionals or not, as IE isn't a thing
anymore), nor default attribute values.
2025-06-29 12:55:34 -07:00
Frédéric Guillot
3de31a1a4d
test(processor): add more unit tests for minifyContent function
2025-06-29 12:53:23 -07:00
jvoisin
560be66147
refactor(misc): Use proper slog.XXX instead of slog.Any
...
This has close to no impact for now, as our slog.Debug/Info/... are leaking
their parameters to the heap, but using proper typing instead of Any allows
to skip some reflection-based computation, making things marginally faster,
and removing the corresponding heap leak.
2025-06-29 12:30:17 -07:00
Ingmar Stein
fcf86e33b9
feat: TLS support for Unix socket listeners
...
This change enables Miniflux to serve TLS over Unix domain sockets.
If `CERT_FILE` and `KEY_FILE` are configured, Unix socket listeners
specified via `LISTEN_ADDR` will now automatically start with TLS enabled,
using the provided certificates. This uses the existing `http.Server.ServeTLS`
method.
If no certificates are provided, Unix socket listeners will continue to
operate as plain, non-TLS sockets.
2025-06-24 21:25:55 -07:00
jvoisin
cbdcf1a56c
Revert "perf(storage): take advantage of entries_feed_id_hash_key in updateEntry"
...
This reverts commit 6527c04307
2025-06-24 19:51:21 -07:00
jvoisin
643b89ec89
perf(storage): take advantage of entries_feed_id_hash_key in updateEntry
2025-06-23 17:59:17 -07:00
jvoisin
8641f5f2a3
refactor(database): drop 3 columns in a single transaction
2025-06-20 16:23:20 -07:00
jvoisin
93b17af78b
refactor(appjs): no need to check if always present elements are always present
2025-06-20 13:16:57 -07:00
Frédéric Guillot
92876a0c61
refactor(http): rename package from httpd to server for consistency
2025-06-20 13:15:13 -07:00
Frédéric Guillot
d62df4e02a
refactor(server): avoid double call to Sprintf
2025-06-20 13:05:21 -07:00
Ingmar Stein
8fa5041c37
feat: Allow multiple listen addresses
...
This change implements the ability to specify multiple listen addresses.
This allows the application to listen on different interfaces or ports simultaneously,
or a combination of IP addresses and Unix sockets.
Closes #3343
2025-06-20 12:46:34 -07:00
Julien Voisin
dc05965895
chore(template): remove X-UA-Compatible meta tag specific to Internet Explorer
2025-06-20 11:59:08 -07:00
jvoisin
109e668ac7
perf(storage): pre-allocate a slice in RefreshFeedEntries
2025-06-20 11:57:04 -07:00
Frédéric Guillot
6d58052504
fix(readability): do not remove elements within code blocks
...
`<span class="hljs-comment"># exit 1</span>` will match the `unlikelyCandidatesRegexp` because it contains the `comment` string.
2025-06-19 21:03:53 -07:00
Matthaiks
491d51c95f
feat(locale): update Polish translation
2025-06-19 18:47:28 -07:00
Frédéric Guillot
db49e41acf
refactor(processor): move FilterEntryMaxAgeDays filter to filter package
2025-06-19 17:56:45 -07:00
Frédéric Guillot
e6b814199b
feat(filter): add EntryDate=max-age:duration filter
...
Example: `EntryDate=max-age:30d` or `EntryDate=max-age:1h`
Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h", "d".
2025-06-19 17:25:19 -07:00
Frédéric Guillot
b0a10f02fd
feat(css): add margin-bottom to input for consistent spacing
2025-06-19 16:35:17 -07:00
Frédéric Guillot
9c05c3c493
feat(filter): merge user and feed entry filter rules
2025-06-19 16:24:57 -07:00
Frédéric Guillot
2a9d91c783
feat: add entry filters at the feed level
2025-06-19 15:15:16 -07:00
Frédéric Guillot
cb59944d6b
refactor(processor): move RewriteEntryURL function to rewrite package
2025-06-19 13:22:29 -07:00
Frédéric Guillot
c12476c1a9
refactor(filter): avoid code duplication between IsBlockedEntry and IsAllowedEntry functions
2025-06-19 12:55:00 -07:00
Frédéric Guillot
bc6ab44ff2
fix(filter): skip invalid rules instead of exiting the loop
2025-06-19 12:36:35 -07:00
Frédéric Guillot
6282ac1f38
refactor(processor): move filters to a filter package
2025-06-19 12:06:30 -07:00
jvoisin
96c0ef4efd
refactor(processor): massive refactoring of filters.go
...
- Use proper variable names for `key=value` strings parts
- Explicitly assign false to the `match` boolean
- Use an explicit `len(parts) == 2` assertion to help the compiler remove
`isSliceInBounds` calls.
- Refactor identical code into a containsRegexPattern function.
- Early exit when parsing the first date fails when using the `Between`
operator, instead of trying to parse the second one.
2025-06-19 11:43:47 -07:00
jvoisin
b139ac4a2c
refactor(youtube): Remove a regex and make use of fetchWatchTime
2025-06-19 11:43:47 -07:00
jvoisin
c818d5bbb8
refactor(youtube): initiliaze two maps to the proper length
2025-06-19 11:43:47 -07:00
jvoisin
e366710529
refactor(processor): remove a useless type declaration
2025-06-19 11:43:47 -07:00
jvoisin
5cff4d7117
refactor(processor): remove a duplication function call
...
As youtubeVideoID is assigned to getVideoIDFromYouTubeURL(entry.URL),
there is no need to call the latter again when we can simly use youtubeVideoID
instead.
2025-06-19 11:43:47 -07:00
jvoisin
f31a784eaa
refactor(processor): refactor common code into a fetchWatchTime function
...
Both nebula and odysee were using the same function to parse time.
2025-06-19 11:43:47 -07:00
jvoisin
7edfcc3cf7
refactor(processor): remove a useless type declaration
2025-06-19 11:43:47 -07:00
jvoisin
fe4b00b9f8
refactor(processor): extract some functions into an utils.go file
2025-06-19 11:43:47 -07:00
jvoisin
46b159ac58
refactor(processor): simplify bilibili processing
...
- Use strings.Contains instead of a regex
- Use strings concatenation instead of a call to fmt.Sprintf
- Use `any` instead of `interface{}`
2025-06-19 11:43:47 -07:00
jvoisin
86c58e11f6
perf(reader): use a non-cryptographic hash when possible
...
There is no need to use SHA256 everywhere, especially on small inputs where we
don't care about its cryptographic properties. We're using FNV as it's the
faster available hash in go's standard library, and we're picking its "a"
version as it's slightly better avalanche characteristics, which are
relevant for small inputs.
This commit has the side-effect of invalidating all favicons saved in the
database, which is desirable to benefit from the resize process implemented in
777d0dd2
2025-06-18 20:28:23 -07:00
