1
0
Fork 0
mirror of https://github.com/miniflux/v2.git synced 2025-06-27 16:36:00 +00:00
Commit graph

148 commits

Author SHA1 Message Date
jvoisin
0e185849b4 Google+ isn't a thing anymore 2024-12-07 16:50:18 -08:00
jvoisin
d0984f29da Simplify isValidTag 2024-12-07 16:50:18 -08:00
jvoisin
902ca63c45 Inline a function and fix a bug in it
The `isAnchor` function's first parameter was always `a`, instead of being
passed `tagName`. As this function is a single line and was only called in a
single place, it can be inlined.
2024-12-07 16:50:18 -08:00
jvoisin
2314500515 Merge two conditions 2024-12-07 16:50:18 -08:00
jvoisin
787d373211 Change the scope of a variable 2024-12-07 16:50:18 -08:00
Julien Voisin
fefbf2c935
refactor(processor): improve the rewrite URL rule regex
- Use `[^"]` instead of `.`, to help the regex engine to determine boundaries,
  instead of having it bruteforce its way to find them
- Use `+` instead of `*`, as empty rules don't make sense
2024-12-07 16:35:51 -08:00
Julien Voisin
bfb429b919
refactor(sanitizer): optimize internal/reader/sanitizer/strip_tags.go
- Use strings instead of doing string->bytes->string
- Use a strings.Builder to build the output
2024-12-07 16:31:48 -08:00
Julien Voisin
331c831c23
refactor(sanitizer): simplify hasRequiredAttributes
This function takes around 1.5% of the total CPU time on my instance, and most
of it is spent in `mapassign_faststr` to initialize the `map`. This is replaced
with a switch-case construct, that should be both significantly faster as well
as pretty dull in term of memory consumption.
2024-12-07 16:30:15 -08:00
Julien Voisin
92a49d7e69
refactor(sanitizer): micro-optimizations of internal/reader/sanitizer/srcset.go
- Pre-allocate a slice
- Inline a local variable
- Remove a superfluous call to `strings.TrimSpace`
- Simplify some conditions via a switch-case construct
2024-12-07 16:27:56 -08:00
Gabe Cook
c3ca603960 fix: load icon from site URL instead of feed URL 2024-12-07 16:06:26 -08:00
telnet23
7e2b50efee feat: optionally fetch watch time from YouTube API instead of website 2024-12-07 16:00:35 -08:00
Gabe Cook
b61ee15c1b fix: feed icon from xml ignored during force refresh 2024-12-07 15:59:49 -08:00
Gabe Cook
30c2e09a56 chore: remove blog.laravel.com rewrite rule 2024-12-03 01:21:42 -08:00
3zero2
c6c71c58b8
feat: add predefined scraper rules for arstechnica.com 2024-11-14 17:47:31 -08:00
AiraNadih
f0fe91172f feat(mediaProxy): update predefined referer spoofing rules for restricted media resources 2024-11-12 19:47:23 -08:00
AiraNadih
b0a3b4d5d9 style(mediaProxy): format with gofmt to pass linter checks 2024-10-30 19:50:12 -07:00
AiraNadih
469f23968e feat(mediaProxy): implement referer spoofing for restricted media resources 2024-10-30 19:50:12 -07:00
Frédéric Guillot
191f3a7ad7 feat(rss): calculate hash based on item title/content for feeds without GUID and link 2024-10-18 18:37:38 -07:00
July
86c0cc61ba
feat: set entry URL to rewritten URL if a rewrite rule is defined 2024-10-13 21:21:28 -07:00
Frédéric Guillot
5c4df786de fix: avoid panic in IsRateLimited() function 2024-10-06 21:34:23 -07:00
Frédéric Guillot
e1050e21b5
feat: take Retry-After header into consideration for rate limited feeds 2024-10-05 22:26:05 -07:00
Frédéric Guillot
f16735fd6d feat: update feed icon during force refresh 2024-10-04 20:51:40 -07:00
Scott Leggett
562a7b79a5 fix: update Last-Modified if it changes in a 304 response
When a server returns a 304 response with a strong validator, any other
stored fields must be updated if they are also present in the response.

This behaviour is described in RFC9111, sections 3.2 and 4.3.4.
2024-10-04 17:47:48 -07:00
Scott Leggett
cb610230d9 chore: update test case comment
The updated comment reflects a better understanding of the RFCs.
2024-10-04 17:47:48 -07:00
Frédéric Guillot
cfe410f202 refactor: split processor package into smaller files 2024-09-22 18:54:19 -07:00
Qeynos
c2ac2bfb83
feat: use Bilibili API instead of web scraping to get video watch time 2024-09-22 18:05:43 -07:00
Pontus Jensen Karlsson
ade412f453 fix: Honor hide_globally when creating a new feed through the api
TestGetGlobalEntriesEndpoint was failing because CreateFeed ignored HideGlobally, this fixes that.
2024-08-12 20:20:44 -07:00
Qeynos
bcbf9f4025
feat: add FETCH_BILIBILI_WATCH_TIME config option 2024-08-01 19:52:31 -07:00
Frédéric Guillot
37309adbc0 fix: do not alter the original URL if there is no tracker parameter 2024-07-25 22:10:28 -07:00
Frédéric Guillot
92f3dc26e4 feat: add support for aside HTML element in entry content 2024-07-25 21:11:37 -07:00
Frédéric Guillot
f6dc952551 feat: add support for base element when discovering feeds 2024-07-25 20:54:51 -07:00
Frédéric Guillot
29387f2d60 feat: implement base element handling in content scraper 2024-07-25 20:36:56 -07:00
Frédéric Guillot
c0f6e32a99 feat: remove well-known URL parameter trackers 2024-07-19 21:35:47 -07:00
Frédéric Guillot
36c25e7689 refactor: simplify Youtube feeds discovery 2024-07-13 12:17:13 -07:00
Frédéric Guillot
cb97d4a1a8 feat: remove YouTube video page subscription finder because meta[itemprop="channelId"] no longer exists 2024-07-13 11:11:50 -07:00
Frédéric Guillot
79ea9e28b5 fix: panic during YouTube channel feed discovery
Regression introduced in commit e54825b
2024-07-13 10:18:15 -07:00
Scott Leggett
bf1c851093 fetcher: use ETag as a stronger validator than Last-Modified
As per the MDN article on HTTP caching:

  During cache revalidation, if both If-Modified-Since and If-None-Match
  are present, then If-None-Match takes precedence for the validator.

  https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching

Previously Miniflux would consider a resource unmodified if the
Last-Modified header had not changed, even if the ETag had changed.

With this commit, Miniflux will consider a resource modified if the ETag
header has changed, even if Last-Modified has not.

This fixes Bug 1 in https://rachelbythebay.com/w/2024/06/11/fsr/
2024-07-02 22:05:49 -07:00
Scott Leggett
c787bb5b48 fetcher: add tests for IsModified behaviour
In particular, add a failing test for the case where ETag changes but
Last-Modified does not.
2024-07-02 22:05:49 -07:00
privatmamtora
1a81866bb9
Add global block and keep filters 2024-07-02 21:03:49 -07:00
JohnnyJayJay
ee5e18ea9f sanitizer: add support for HTML hidden attribute
This commit adjusts the `Sanitize` function to skip tags with the
`hidden` attribute, similar to how it skips blocked tags and their
contents.
2024-06-21 14:00:40 -07:00
Ztec
9f3a8e7f1b Request builder: Allow the use of insecure TLS ciphers when Allow self-signed or invalid certificates is used
Some server on the wild are badly configured. Either by mistake or lack
of maintenance. Safe and unsafe Ciphers change overtime based on new
discoveries.

This proposition will include considered unsafe ciphers when `Allow self-signed or invalid certificates` is used.
It could be put into a separate option but, I felt this could fit in.

fix #2671
2024-06-13 20:23:37 -07:00
Ztec
e54825bf02 Improve YouTube page feed detection
In order to be more resilient to YouTube URLs variation and
to address this feature_request: https://github.com/miniflux/v2/issues/2628
I've reworked a bit the way the YouTube feed extraction is done.

I've kept all the `FindSubscriptionsFromYouTube*` in order
to keep all the existing unit tests as-is ensuring little to no
regressions. By doing so, I had to call twice `youtubeURLIDExtractor`.
Small performance penalty for peace of mind in my opinion.

`youtubeURLIDExtractor` is made in a way only one kind
of page can be detected at a time. This mean I can
solve the "video in a playlist" feature_request
by prioritizing the playlist ID over the Video ID

Also, by using `url.Parse()` to get ids, it's safer
to url mangle and variation. The most common variation
being the `t=42` parameters that start the playback
at a given position. Previously, this kind of url
would not be detected as "YouTube URL".

I deliberately ignored the url parsing error
to keep previous behavior (skip the YouTube analysis and follow with the other analysis)

I also tried to keep debug logs the same as before as much as I could.

I manually tested all the YouTube cases (video,channel,playlist)
and they all work as expected except for the video. But this one
does not work either on main. The `meta` html tag that was searched for
does not seem to exist anymore.

fix: #2628
2024-06-13 20:18:47 -07:00
x
839fc3843a Add pitchfork.com scraping rule 2024-06-10 21:08:59 -07:00
x
0bab8fac8e Update theverge.com rewrite rule: fix duplicate image
See: https://github.com/miniflux/v2/issues/1979
2024-06-10 21:08:59 -07:00
Ankit Pandey
b68b05c64c reader/processor: error out for improper rewrite regexp
It's possible to specify a rewrite regex that validates but doesn't compile such
as:

    rewrite("(((unmatched-capture-group"|"rewrite)))")

In case we encounter one, exit early instead of letting the server panic.
2024-06-01 10:37:02 -07:00
Zhizhen He
ae432bc9c6
reader/readingtime: fix incorrect package name 2024-05-21 18:12:24 -07:00
Jan-Lukas Else
a33b1adf13 Add description field to feed settings
This adds a new "description" field to the feed settings. This allows to
save custom description regarding a feed. It is also exported and
imported as "description" in OPML.
2024-05-06 15:40:36 -07:00
fin444
a631bd527d options: add FETCH_NEBULA_WATCH_TIME 2024-05-02 16:30:01 -07:00
Frédéric Guillot
fb075b60b5 reader/processor: minifier is breaking HTML entry content 2024-04-23 20:31:52 -07:00
Frédéric Guillot
2c4c845cd2 http/response: add brotli compression support 2024-04-19 12:16:49 -07:00