Question: [QUESTION] "Could not find prepared statement with handle 1" - probably, it was fixed?

The 12.7 preview release is available. The next stable release to include this will either be the 12.6.2 hotfix and/or the 12.8 GA release. We have not finalized a date for the hotfix, but the GA can be expected at the end of July (July 31).

[QUESTION] "Could not find prepared statement with handle 1" - probably, it was fixed?
## Question

Random exception: "Could not find prepared statement with handle 1", mssql-jdbc-11.2.3.jre11.jar
Probably, it was fixed in latest versions?

Stack:
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Could not find prepared statement with handle 1.
 at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:265) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1676) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:620) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:540) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7620) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:3916) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:268) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:242) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQueryInternal(SQLServerPreparedStatement.java:473) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getResultSetFromStoredProc(SQLServerDatabaseMetaData.java:360) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getResultSetWithProvidedColumnNames(SQLServerDatabaseMetaData.java:374) ~[mssql-jdbc-11.2.3.jre11.jar:?]
 at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getIndexInfo(SQLServerDatabaseMetaData.java:1162) ~[mssql-jdbc-11.2.3.jre11.jar:?]
## Relevant Issues and Pull Requests

Seems not to be fixed recently - I get the same errors randomly after some time since switching from 12.4.2.jre8 to 12.6.1.jre8

com.microsoft.sqlserver.jdbc.SQLServerException: Could not find prepared statement with handle 1.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:261)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1752)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:675)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:594)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7739)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:4384)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:293)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:263)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:509)

hi @smesh

this looks like it might be related to #2356 could you please tell us what your connection string is?

Hi @lilgreenbird ,
Connection url is: jdbc:sqlserver:// ;authentication=SqlPassword;DatabaseName= ;trustServerCertificate=true;SelectMethod=cursor

we have another issue relating to this that we are working on, we think it's related to the cache not being cleared. Can you pls try to disable prepared statement handle caching and see if that resolves your issue? if so, that's a temporary workaround for the issue for now.

i.e. add `disableStatementPooling=false;statementPoolingCacheSize=0` to your connection string

Sorry for not replying, we could not try workaround in the production.

Erroneously inserted rows with statement pooling enabled and when a Connection gets re-used from the ConnectionPool
## Driver version

 com.microsoft.sqlserver
 mssql-jdbc
 12.6.1.jre11

## SQL Server version

Microsoft SQL Server 2019 (RTM-CU18) (KB5017593) - 15.0.4261.1 (X64)
 Sep 12 2022 15:07:06
 Copyright (C) 2019 Microsoft Corporation
 Developer Edition (64-bit) on Windows 10 Pro 10.0 (Build 19045: )

## Client Operating System
Windows 10 Pro

## JAVA/JVM version
Java 17

## Problem description
There are some duplicate inserts and some missing entries when a custom connection pool is being utilized and server side statement pooling is enabled (using the setDisableStatementPooling flag). When the connection is re-used from the connection pool and the flag `setDisableStatementPooling is set to false` this issue seems to happen.

To test this instance, I was able to simulate a very basic stand alone as below :
Please see the following test program, where I intentionally return the previously created connection object from the _userToConnectionsMap cache map for subsequent runs.

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Collection;
import java.util.Iterator;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.locks.ReentrantLock;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.sql.PooledConnection;

public class SQLConnectionPool {

 final private ReentrantLock _connLock = new ReentrantLock();
 final private Map _userToConnectionsMap = new ConcurrentHashMap();
 static String tableName = "batchTest";
 /**
 * Crate a new connection
 *
 * @return the new created connection
 * @throws SQLException When fail to create a new connection.
 */
 private Connection createNewConnection(final String username, final String password) throws SQLException {

 com.microsoft.sqlserver.jdbc.SQLServerConnectionPoolDataSource msDs = new com.microsoft.sqlserver.jdbc.SQLServerConnectionPoolDataSource();
 msDs.setUser(username);
 msDs.setApplicationName("SQL Server Duplicate Row");
 msDs.setPassword(password);
 msDs.setServerName("<>");
 msDs.setPortNumber(0);
 msDs.setDatabaseName("<>");
 msDs.setInstanceName("<>");
 msDs.setEncrypt("false");
 msDs.setDisableStatementPooling(false);
 msDs.setStatementPoolingCacheSize(20);
 msDs.setServerPreparedStatementDiscardThreshold(20);
 msDs.setSendTemporalDataTypesAsStringForBulkCopy(false);
 // ... create the ACTUAL new connection...
 PooledConnection actual = msDs.getPooledConnection(username, password);
 // ... cache it...
 Collection conns = (Collection) _userToConnectionsMap.get(username);
 if (conns == null || conns.isEmpty()) {
 conns = new CopyOnWriteArrayList();
 _userToConnectionsMap.put(username, conns);
 }

 conns.add(actual);
 return actual.getConnection();
 }

 final public Connection getConnection(final String username, final String password)
 throws SQLException, InterruptedException {

 if (username == null) {
 throw new NullPointerException("The username cannot be null.");
 }
 if (password == null) {
 throw new NullPointerException("The password cannot be null.");
 }

 // Acquire a lock...
 _connLock.lock();
 // Declare the connection to be returned...
 Connection conn = null;
 try {

 conn = getAvailableConnection(username);
 // If none found...
 if (conn == null) {
 conn = createNewConnection(username, password);
 }
//
 } finally {
 // Ensure that the lock is released...
 _connLock.unlock();
 }
 // Return the connection...
 return conn;
 }

 private Connection getAvailableConnection(final String username) throws SQLException {
 // Get the connections for the specified user...
 Collection conns = (Collection) _userToConnectionsMap.get(username);
 // If none found, then NO connections are available...
 if (conns == null || conns.isEmpty()) {
 return null;
 }
 // ... otherwise, check to see if any of the connections are available...
 boolean isAvailable = false;
 final Iterator iterator = conns.iterator();
 PooledConnection pconn = null;
 while (iterator.hasNext()) {
 pconn = (PooledConnection) iterator.next();

 return pconn.getConnection();
 // Is this connection available? ignore this check for this sample test

 }
 return null;
 }

 public static void main(String[] args) throws SQLException, ClassNotFoundException, InterruptedException {

 Logger logger = Logger.getLogger("com.microsoft.sqlserver.jdbc.Statement");
 logger.setLevel(Level.FINER);
 Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");

 SQLConnectionPool pool = new SQLConnectionPool();

 try {

 Connection conn = pool.getConnection("<>", "<>");
 try (Statement stmt = conn.createStatement();) {

 String dropSql = "if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[" + tableName
 + "]') and OBJECTPROPERTY(id, N'IsUserTable') = 1) DROP TABLE [" + tableName + "]";
 stmt.execute(dropSql);

 String createSql = "create table " + tableName + " (c1 int, c2 varchar(20))";
 stmt.execute(createSql);
 }
 for (int j = 0; j ", "<>");
 conn.setAutoCommit(false);
 try (PreparedStatement pstmt = conn.prepareStatement("insert into " + tableName + " values (?, ?)");) {
 for (int i = 0; i < 3; i++) {
 pstmt.setInt(1, i);
 pstmt.setString(2, "test" + i);
 pstmt.addBatch();
 }

 long start = System.currentTimeMillis();
 pstmt.executeBatch();
 conn.commit();
 long end = System.currentTimeMillis();

 System.out.println("Finished. Time taken : " + (end - start) + " milliseconds.");
 }
 conn.close();
 }
 } finally {
 // nothing for this test
 }

 }
}

This program results in incorrect entries being inserted to the test table with last item of the batch being repeated and the first item of the batch missed out.

## Expected behavior
Caching and connection pool shouldn't lead to incorrect insert to the table. Should see no duplicates and no missing rows.

## Actual behavior
At the end of the stand alone program run, could see there are (2*j) - 1 = 9 rows instead of 5 rows for column c1 : 2 and (j-1) = 4 entries missing for column c1 : 0

image

## Error message/stack trace
With FINER logging enabled for SQL driver , could see the following exception : `SQLException: com.microsoft.sqlserver.jdbc.SQLServerException: Could not find prepared statement with handle`

image

## Any other details that can be helpful

This issue does not happen when I comment the call to `sp_reset_connection` in **SQLServerPooledConnection.java**
image

## JDBC trace logs
SQL Profiler data

image

hi @PriyadharshiniP

thanks for the repro we can see the issue will be implementing a fix for this in the upcoming preview.

Thanks for looking into this.

1. Additionally, it appears that the following modifications to the `SQLServerPreparedStatement` : by resetting the `batchParam` value to fetch the metadata of first row(0) instead of the last row seems to prevent duplicates from happening in this case.
But not entirely sure of the consequences.
Please have a look as it may help you fix this issue :
image

2. I'm also not clear why, when we re-use the same connection from the pool, a `Could not find prepared statement with handle exception is thrown`. I would really appreciate any clarifications to assist me better comprehend statement pooling.

I believe this is happening cos the cache is messed up. If I clear the preparedStatementHandleCache in resetPooledConnection() then this works. I'm just doing some testing now.

I appreciate you committing the fix.
It appears to prevent duplicates from happening by creating a new prepared statement handler for every usage of same connection from the connection pool. But I don't fully understand this fix.
I am aware that this statement pooling is per-connection and not server-wide.
But the prepared statement handler should be retained for re-using the connection having the same user name and password from pool : SQLServerPooledConnection , is it not ?
If statement caching was implemented and a connection pool was in use, it seems to be a waste in both client and server side and no real performance improvements with statement cache?
Please clarify.

Statement pooling is per connection, yes, but sp_reset_connection (resetPooledConnection) clears any prepared statements in the session on the server (among many other cleanup tasks). That's why you see the _Could not find prepared statement with handle_ error. Clearing the cache would keep the driver from re-using invalid prepared statement handles. But I'd be worried that the driver is executing some of the batch or consuming/flushing results on the wrong side of the pool use boundary. ExecuteBatch() should be executing and consuming all the results from the server immediately and I wonder if it's leaving some hanging until the next statement execution (which is happening after the connection goes back into and then comes out of the pool again).

Thanks for clarifying that the statement pooling is per connection and cannot be shared even when the connection is used , goes back and is re-used from pool.
It appears that the un prepare call is not being initiated explicitly, and presumably, there can still be a performance improvement if statement pooling and pooling are enabled.

With the fix under this issue, it seems to have fixed the corrupt client side prepared statement cache by clearing it when the connection gets re-used from the pool.
But I am still not sure, if there can be some duplicates or entries from a batch mistakenly inserted due to any other scenarios where there is a corrupt prepared statement cache or no server prepared handle with a retry. I cannot replicate such a scenario yet with 8179 and 586 exceptions. However it is in the exception handling, I think it may be possible.
The row in batchParam ought to have also been reset to the index of numBatchesPrepared rather than the last row if the retryBasedOnFailedReuseOfCachedHandle method returns true, right? I'm saying that there may still be instances where this exception stack trace occurs, and that the batch should be correctly retried.
Please advise.
image

yes we agree there may be other issues there we will take a closer look into this when we can, thanks

Hi @lilgreenbird
Thanks for fixing. May I know when a stable release (12.7.0) for this be available ?


Question: Don't pass test data to `.predict()`?

This is a very useful mental framework. We should keep in mind that a refactor of the autoregressive model is imminent, after which (1) and (2) will be cleanly separable just like for the Hill model. The soon-to-be-refactored AR model will have parameters which govern the true latent incident uptake from week to week, and separate parameters which govern the observation process. Thus, in both cases, we will be able to use the $n$ posterior samples generated during fitting to produce retrospective estimates or future forecasts of true latent uptake, and then "add" observation uncertainty to that, using $k=1$ draws per posterior sample as you've suggested. Does this sound correct, @afmagee42?

Don't pass test data to `.predict()`
@eschrom We talked about the possibility for `.augment_data()` to leak information about the future into models. But I'm realizing that `.predict()` explicitly takes the *test* data? Presumably this is something about target dates, but it's something we should change because (1) this is incompatible with doing prospective forecasts and (2) it's a real bad smell w.r.t. leakage.

You are correct - `predict()` wants the test data so that it can adjust the user-specified target dates to match the closest target dates in the test data, during retrospective forecasting.

Note that:
1) When making prospective forecasts, the `forecast.py` script sets `test_data` to `None`, which is passed to `predict()` and is sensibly interpreted to mean that the user-specified target dates need not be adjusted at all.
2) Inside `predict()`, `test_data` is _only_ passed to `augment_scaffold` for the purpose of (possibly) adjusting target dates; `test_data` has no influence on the uptake values that are predicted for those target dates.

Maybe the appearance of passing `test_data` to `predict()` is not ideal, but I think the substance of it is fine.

Agree, I don't have any concerns about the current work

But I do think it would be better to refactor (at some point) so that predict takes target dates, rather than something that looks like data about the future

This is pertinent to the discussion that evolved in #110, and in particular sounds like an argument for Option 3 in URL

Separate fitting and forecasting seeds
I think this is worth a separate discussion-- I think we might want separate fitting and prediction keys

Like, I think it's fair for us to fit the model a small number of times (1? 3?) but then we'll want to draw a number of trajectories (100? 1000?).

I want the same fitted model to produce the same trajectory when given the same input key.

There are some nice ways to generate child keys from a parent key.

_Originally posted by @swo in URL

To the extent that prediction requires drawing new values, then from first principles I think having it be key-configurable is a good idea?

But I think we're in a danger zone for confusion here and need to be careful about what we mean and what's being separated from what. There are two distinct data-conditioned sets of things which we can refer to with prediction:
1. Draw observations from the observation model, this is the posterior predictive distribution. NumPyro provides this with `Predictive` (which also includes the prior predictive, but that's not data-conditioned).
2. _Forecasting_ predictions; asking about what will happen in the future.

These aren't always neatly factorizable.

As of #125 with the Hill model, we have a model where they can be factorized. That model has a deterministic uptake curve and a model for how reality gets observed from it. For the Hill model, predictions about the latent true uptake in the future, that is (2) are deterministic given posterior samples, but that needn't be true in general. And for any time period, future or not, given the latent uptake (1) involves a "new" RNG call, in that if all you have are posterior samples, you can't just get this for free.

The autoregressive model, as currently implemented and as I understand it, hides this distinction by saying that what we've seen _is_ the uptake, so for the future (1) + (2) are the same, and in the past they're inapplicable.

I suggest that the general easy mental model for thinking about this all is that the number of posterior predictive samples which is _natural_ for $n$ samples from the posterior distribution is
1. $k n$, where we usually have $k = 1$ and draw one new observation per posterior sample. This is what stan does. (We can also "thin" if we need to here, and have $k 1$ and draw more.)
2. $n$. This works both for models which do not have future randomness given the posterior (Hill) and which do (autoregressive). (Technically we could have more samples for models with additional randomness, but I don't think it's worth the headache.)

Yep. I'd say that in general the order of operations is relatively flexible, but my understanding of NumPyro is that (2) as post-processing is the natural fit. (In stan, it's in-MCMC with `generated quantities {}`.)

Which brings up one last consideration here, while there's in-progress refactoring: should (1) be done as part of the posterior sampling, or as post-processing. That is, should MCMC produce, for each sample, a latent true trajectory for _all times_ we will ever want to know the value, whether or not we have data for them?

My inclination is to say that we should do it as part of MCMC, if at all reasonable. For the cost of carrying around some extra variables in MCMC, it simplifies the post-processing greatly because all models only need to be able to draw from their likelihoods/observation models. It should also tend to make shapes more consistent, as a posterior for the same season will always have all parameters of the same length.

Producing a latent true trajectory for all times we will ever care about could be a little tricky. We want the user to specify the exact time frame over which to make forecasts. For example, a user could request uptake to be forecasted weekly from March 3 2025 to June 2 2025, or from March 4 2025 to June 3 2025. These will differ slightly. So to be safe, each model would need MCMC to return a latent true trajectory with predicted uptake on every single day within each season.

 This is fine for the Hill model, but the AR model currently uses "incident uptake one week ago" to predict "incident uptake this week." More precisely, the AR model uses "the daily average uptake last week" to predict "the daily average uptake this week." As such, each observed data point predicts the next, without being fooled by the occasional non-7- (usually 6- or 8-) day interval between observations. So the AR model's notion of single days (rather than weeks) isn't as explicit as it would need to be, in order to do what you're suggesting.

This brings us to a conversation we've had in another channel - maybe the AR model should account for slight variations in the inter-observation intervals differently, instead of this "daily average" business I've been using. Perhaps the AR model should use "incident uptake yesterday" to predict "incident uptake today." In this setting, a lot of data is "missing" - we have observations roughly once every 7 days rather than once every day - but it's simple enough for a model to generate predictions of uptake daily and yet to only be compared to observations weekly. Is this part of the AR refactor you have in mind, @afmagee?

I am of several conflicting minds here.
- I was under the impression that we really did have weekly data (with the occasional multi-week skip at the beginning) in which case a shift of a day's difference only should matter if that moves the start date/end date across a week boundary, so we shouldn't have to worry about daily at all.
- Make it daily and don't worry about missing data too much. I agree that computing the weekly likelihood from daily data isn't that big a deal. On the other hand, this introduces a _lot_ of extra variables to sample for stochastic models, and worse off, they're going to be highly correlated in a block structure (the weekly data means that increasing one day's uptake will favor decreases in at least one other day).
- Reframe the problem and change the model specs a smidge. Make the user say (1) when the observations are and (2) when they want forecasts for. No restrictions on when those are, other than "within the season." Then make all models have to be able to deal with an irregular grid of observations.
 - This is clearly not a problem for something deterministic that lives in continuous time, like a Hill model.
 - For autoregressive models, some re-thinking is needed, but doable. Mostly it means that you'll not just have a mean model but also a variance model. Consider a model where the only thing that matters is the last observation's uptake. The mean model is $\mu(t\_{i + 1}) = \beta u(t\_i)$. The variance model is $\sigma(t\_{i + 1}) = \sigma^2 (t\_i - t\_{i - 1})$. When all points are evenly spaced, we can remove the variance model for a constant variance. But when they're not, it says that while last time point's uptake is predictive of this time point's, the effect diminishes with distance. We'd have to work out some details on other bits, and make sure it all plays together. But again, it's doable.

The irregular grid option is attractive for
- Allowing us to get the shape fixed in advanced while trimming some extra baggage (daily predictions we'll never use)
- Including "just do daily" as a special case.

The tradeoff being the bit of extra math in some cases, and that technically I think it makes some stochastic models inapplicable and/or wrong/incoherent. Because what we're really doing is approximating integrals/sums over the missing data, and that doesn't always add up to a nicely computable distribution on the next time point. But if we're thinking about an underlying discrete daily model, and everything is Gaussian, things may well integrate out such that there is an easy "right" thing that looks like the above. I can convince myself it's "right" for the model above when $\beta = 1$, but I'm not _quite_ sure it's right otherwise. Anyways, whether or not it's technically incoherent as a stochastic process doesn't have the biggest bearing on whether it works as a prediction tool.


Question: [FP]: jackson-databind-2.13.2?

One final thing: I also went ahead and released `jackson-bom:2.13.2.1` just so that Gradle Module Metadata issue for `jackson-databind:2.13.2.1` SHOULD be fully resolved. So while there is also `2.13.2.2` of `jackson-databind` (with actual already valid GMM) it seems better not to leave 2.13.2.1 with dangling reference to non-existing module info.

[FP]: jackson-databind-2.13.2
### Package URl

pkg:maven/com.fasterxml.jackson.core/jackson-databind@2.13.2

### CPE

cpe:2.3:a:fasterxml:jackson-databind:2.13.2:*:*:*:*:*:*:* and cpe:2.3:a:fasterxml:jackson-modules-java8:2.13.2:*:*:*:*:*:*:*)

### CVE

CVE-2020-36518

### ODC Integration

_No response_

### ODC Version

7.0.1

### Description

From URL

When reporting this CVE for jackson-databind 2.13.2 it seems like a false positive.

Hello, :wave:

This does not looks like a false positive to me. The vulnerability has been acknowledged by the jackson-databind contributors and they are actively working on it. A fix has been integrated in version `2.13.2.1`, `2.13.2` is considered to be impacted. Please see URL

Hi all 👋 please note that one more patch, `2.13.2.2`, has appeared.

Optimize `UntypedObjectDeserializer` wrt recursion [CVE-2020-36518]
**EDIT**: related to to CVE-2020-36518 (see URL

**EDIT**: Fix included in

* 2.13.2.2 (see URL
 * use with `jackson-bom` version `2.13.2.20220328`
 * NOTE: ALSO included in `2.13.2.1` but there is a problem in Gradle Module Metadata declaration which is why `2.13.2.2` was released -- if not using GMM can use either
* 2.12.6.1: (see URL
 * use with `jackson-bom` version `2.12.6.1.20220326`

-----

Current implementation `UntypedObjectDeserializer` is relatively expensive for deeply nested Object and Array values as it uses recursion even for "vanilla" case (one where there are no custom `List`/array or `Map` deserializers).
In practical terms it is possible to exhaust typical modest JVM heap sizes with documents having about ten thousand levels of nestings.

*NOTE*: specifically this ONLY APPLIES if the target type is "untyped" or generic `Collection ` / `Map ` -- it DOES NOT APPLY to cases where target is POJO (except if POJO itself has "untyped" property or properties).

Similar issue was already solved wrt `JsonNode` (see #3397), included in 2.13.0; this might show a way to approach this problem: by replacing simple recursion with iteration, either completely or at some inner levels.

Also note that it may ultimately be necessary to have lower-level constraints for streaming parser too, see: URL

Ideally it should be:

1. Possible to handle at least tens of thousands of levels of nesting (100k should be processable with 256M heap, say)
2. Have streaming level limits that -- by default -- block documents with more than limit we deem safe (less than 100k -- perhaps 10k or something, to be determined).

This issue is specifically about (1) as (2) is about `jackson-core`.

Looks like nesting of about 2000 works, 3000 fails, with default settings for both Arrays and Objects.

FWTW, similar issues does NOT affect `JsonNode`, as of Jackson 2.13.0 and later on.
Should have created separate issue to point to, but iterate version of `JsonNodeDeserializer` was merged in at around March 2021.
Would be great to achieve the same here for 2.14.

@cowtowncoder , could you please clarify if the same fix is planned to be backported to the versions `2.12.x` and `2.11.x` of Jackson?

@meier-th Very unlikely that I would try to backport this to anything prior to 2.14.0. The new implementation probably has non-trivial chance of regression and this functionality is widely used. Note that no implementation yet exists, but I fully intend to work on this to try to get it done for 2.14.0 (or someone else could beat me to it).

@meier-th I'm sure that the same issue existing in `2.12.x` and `2.11.x`. May I know the fixing plan ? if the issue can only be fixed by upgrading to `2.14.x`, when do you plan to release this version ?

Oh fucking great. Someone decided to file an CVE for this one.

Surely teaches me to file issues on things I want to work on -- and then some arsehole going to file an CVE.

I recommend that you simply throw some error if the nesting is to deep to cope with. You could improve the nesting depth later on, but you (and we users) will get rid of this annoying cve.

You could just add a try/catch around UntypedObjectDeserializer's entire deserialize method. The catch clause could just be:

 } catch (StackOverflowError soe) {
 throw new JsonMappingException("JSON is too deeply nested.", soe);
 }

Then instead of throwing an undeclared/unchecked runtime error (StackOverflowError), it throws a declared/checked JsonMappingException. That should mean anyone calling the function is actually catching the exception and hopefully doing something appropriate instead of letting their application fail in an unexpected way.

I tested the above on the 2.13 branch - is it simple enough to just release that as a 2.13.3? I wasn't sure whether it would actually be appreciated or not so I didn't make a PR, and I'm not sure if there's other spots (or a better spot) to check for the StackOverflowError at. But I can confirm I did it locally and that it compiles. The currently failing test could be changed to expect a JsonMappingException instead of expecting an object to be returned.

First of all: thank you everyone for your ideas, input, help. Much appreciated!

@wsdng That is one possibility, but due to statelessness of deserializers requires a bit of refactoring. It also would be better to add limits at streaming level (there is a separate issue there) but I do agree that there's timing issue where ideally this would be tackled first.
Since I have the general idea -- of not having to use nesting counters -- from earlier `JsonNodeDeserializer` work all I need is time to work on this.
I did consider simple (conceptually) counter approach earlier and only abandoned that since solving it in (IMO) better way by avoiding recursion. This is (I think) better esp. eventually when we get streaming level general limits.

@TaylorSMarks unfortunately I don't think that approach is workable: by the time SOE is thrown the damage is done -- all catching would do would be to give slightly friendlier exception message.
While that can be useful in itself (and POJO (de)serializers do this, I think) I don't think it helps us here, unfortunately.

Hey, thanks everyone and @cowtowncoder for this library.

I just wanted comment that; unfortunately, our nexusiq instance started to fail our builds by this issue as indexed 7.5 HIGH severity. I think the only solution is to waive this CVE for now.

Btw, still planned in 2.14? Not possible to patch only this issue in 2.13.3?

@cowtowncoder - I have approval to spend a day of company time on fixing this. Our company uses Black Duck for security scanning, which is flagging all versions of Jackson-databind for having this issue.

I'm hoping for a solution that's simple enough to implement in a few hours, factoring in that I'm not particularly familiar with this project (I wrote a few custom deserializers a few weeks ago - that's the most exposure I had prior to downloading/compiling/running the tests so that I could make the PR last night.)

Fixes I can think of, in order of how simple they'd be to implement/test:

1 - Catch the SOE and throw a checked exception. This should improve stability of the code calling it, but it has the issues you mentioned.

2 - Add a counter, maybe to the DeserializationContext? Everytime we enter an array or object, increment the counter. When we return, decrement it. If the counter exceeds a predefined depth, throw an exception. What's a reasonable default for the max depth? 20? 100? 200? In the exception message mention how to configure it to allow for more deeply nested objects. This has the same benefits as what I did for the PR, and now additionally it means we intentionally limit how much resources are consumed by a malicious payload. The downside is there's some cost to having to keep track of how deep we are. The cost seems acceptable to me.

3 - Don't make Vanilla.mapArray and Vanilla.mapObject separate methods - just include them straight in Vanilla.deserialize. I question the value of even mentioning this. It's hideous and decreases maintainability. The only positive thing to say about it is it doubles the maximum depth of the payload that can be handled before it throws a Stack Overflow Error.

4 - Removing recursion. I suspect there's a lot of existing code that I haven't looked at that I'd need to understand and rework to do this. I think it's worth pointing out that, like #3 above, this doesn't really fix the issue, though. You still have the potential to consume all available memory and then throw an OutOfMemory exception.

I'm pretty sure the second of the options I listed is best from a security perspective. As soon as we realize it's not a serious payload, we're rejecting it, minimizing wasted resources and throwing a checked exception which the calling code will know they have to handle somehow. I'd probably also leave what I already did yesterday for the PR (the first of the proposals)... leaking a runtime exception if they set the parameter too high and they received a payload that's too big doesn't seem great.

Ok, starting in reverse order.

2.13.3. I guess due to high visibility, it may be necessary, despite non-trivial regression risk.
Release takes 2-4 hours for me; Jackson currently is not part of my daytime job. So it's weekend voluntary work for me, essentially. If and when CVE was solved I'd say 3-10 days after that.
Well, except... this would qualify for a micro-patch, 2.13.2.1. That can be released much faster since it is only 2 repos (Jackson-databind and Jackson-bom). I think that'd be the way to go.

As to suggested approaches, my initial quick thoughts:

1. Not worth pursuing, IMO.
2. Something I have considered and may be viable -- selecting default value is problematic indeed; making it configurable even bigger issue since there's no clean API to extend. AND if this is temporary... in fact I would leave it non-configurable altogether at first, waiting for better solution IF this was pursued
3. Yeah just doubling does not really seem like worth the effort.
4. This is the approach I would take (and am planning to take): have a look at `JsonNodeDeserializer` which did this for 2.13; can contrast with earlier (2.12) implementation.

As to (4) "not fixing", I agree and disagree. Correct, fundamentally one can still run out of resources with big enough input. But to me the initial point is that there is difference between say, "only 2000 levels" (document of 2000 characters) and something like "only 1 million levels" (megabyte document). So this fix would change the nature of problem; you cannot just send a small document, you need to be able to send a very much bigger one.

But I think bigger reason why I think this makes sense is that I think that fundamentally limits for document aspects should go through this:

 URL

so that the ultimate protection is via global document limits ("maximum nesting: 10,000 levels"), but we also make default deserializers more robust to let us keep the default document limits rather loose.

So: I'd go for (4), similar to handling of `JsonNode`: and for 2.14 try to get general document limits for streaming parser.

@cowtowncoder - This PR has changed from just catching a Stack Overflow Error and rethrowing a JsonMappingException to instead tracking the nesting depth and throwing if it's "Too Deep".

256 was chosen as the max depth. It's mostly arbitrary. I looked into the maximum folder depth allowed on different platforms - it's less than 130 on Windows, less than 512 on macOS. I don't hear people complaining that they can't nest their folders any deeper on Windows, so... I figured 256 was a safe maximum depth for json.

The unit tests were updated. It'll verify that 250 nested arrays or objects will deserialize properly. It verifies that 300 nested arrays or objects throws a JsonParseException.

To be honest, I don't see how this is a CVE for only Jackson when both Gson and JSON-B have the same issue.

I've created a small test program that creates a simple JSON string with only nested objects for any given depth, then tries with Jackson, Gson and JSON-B until there is a StackOverflowError.

 public static void main(String[] args) throws Exception {
 testJackson();
 testGson();
 testJson();
 }

 private static void testJackson() throws IOException {
 final int maxDepth = 10_000;
 for (int i = 2000; i map = new ObjectMapper().readValue(json, Map.class);
 map.size();
 } catch (StackOverflowError e) {
 System.out.printf("Jackson: %d%n", i);
 return;
 }
 }
 System.out.printf("Jackson: no failure for depth %d%n", maxDepth);
 }

 private static void testGson() {
 final int maxDepth = 10_000;
 for (int i = 2000; i map = new Gson().fromJson(json, Map.class);
 map.size();
 } catch (StackOverflowError e) {
 System.out.printf("Gson: %d%n", i);
 return;
 }
 }
 System.out.printf("Gson: no failure for depth %d%n", maxDepth);
 }

 private static void testJson() {
 final int maxDepth = 10_000;
 for (int i = 2000; i map = JsonbBuilder.create().fromJson(json, Map.class);
 map.size();
 } catch (StackOverflowError e) {
 System.out.printf("Json: %d%n", i);
 return;
 }
 }
 System.out.printf("Json: no failure for depth %d%n", maxDepth);
 }

 private static String createJson(int depth) {
 StringBuilder sb = new StringBuilder(10000);
 for (int i = 0; i < depth; i++) {
 sb.append("{\"m\":");
 }
 sb.append("{}");
 for (int i = 0; i < depth; i++) {
 sb.append('}');
 }
 return sb.toString();
 }

Unsurprisingly, it's not just Jackson that fails but the others as well. Gson fails later, but it still fails. JSON-B fails at around the same depth.

I can not understand why this becomes a CVE with a "high" rating.
DOS attack, really?
Such a StackOverflow occurs after about 500ms when I try this here. Means an attacker would get the error after that time, which seems not to be too high for me.
Minimal effort on flow control on the http level (which you need to have in place anyway!) is enough to mitigate this.
Or do I miss something here?

@schlm3 You can also exhaust some heap this way. However that will probably just lead to an error for the attacker also, not actual DoS for other users. I guess an attacker could open a connection, consume a large amount of memory in UntypedObjectDeserializer, and then keep it open for a while to put pressure on the heap that affects other users. But yes, the "high" CVSS availability impact that justifies the "high" CVSS for this vuln seems a bit far-fetched. (easy exploitability is true, though ofc most users aren't using the `UntypedObjectDeserializer` anyway so are not impacted)

The "high" rating for this vuln is more an issue with the limited info that can be crammed into a single CVSS score.

@schlm3 I am with you here, I don't quite understand the high rating. And like someone mentioned (I think? but can't see a reference), many other JSON libraries have similar challenges -- so theoretically similar CVE should be filed against GSON, Genson, fast-json, for relevant types (disclaimer: I have not tested personally which ones have similar problem, some might have checks). But the reason Jackson gets the first dibs is due to our popularity: the potential attack vector is bigger, so that's somewhat reasonable I guess. Underhanded compliment if you wish :)

I have heard that some of the ratings are also auto-generated; or, almost as bad, just using submitters original suggestion with little validation. I sort of understand this in that fully grokking most important aspects is a lot of work, and the torrent of reports is high. It's bit like challenges patent application clerks face in evaluating if an app is actually novel invention -- not enough time to do proper searches; or as in here, dig in the code.

@abdullahali99 there is a plan (see PR #3416) which for practical reasons probably must be micro-patch like `2.13.2.1` of `jackson-databind` (along with matching `jackson-bom`).
I hope to get it out some time this week. Really wish system did not cause these stupid fire drills for CVEs of dubious severity rating, but here we are :-)

Merged the fix in for 2.13; left out for 2.14 and master (3.0) for now. Getting late, will try to get release out tomorrow.

@cowtowncoder I added that as a comment here first, but then decided that URL was a better place to place that comment, so I removed it here.

I have reproduced this issue with Gson and JSON-B; the code snippet in that comment can easily be adapted to also test others.

@cowtowncoder there is contradictory guidance in the market about your fix release version. Could you please clarify "Merged the fix in for 2.13..." Is 2.13.* vulnerable? What is the fix version number?

@jaime-castells It has been merged into the 2.13 *branch*, but there has not been a 2.13 *release* yet. The release with the fix will be 2.13.3.

Out of curiosity, do you think there will be a 2.12.6.1 with the fix?
Not sure what your threshold is for micro-patches on 2.12, or if the 2.13 fix will even backport easily.
 URL

I'd really prefer not to have to backport this, but... based on past experiences I have a feeling I will cave in and do backport :-D
2.12 branch is officially still open, unlike earlier ones. And I suspect `UntypedObjectDeserializer` can actually be merged without problems.

@cowtowncoder thank you in advance for your time on this. How this CVE came about is a little ridiculous, and the severity it was given is not very defendable.
@TaylorSMarks thank you for contributing URL

My one and only reason in replying: Open source is not a thankless job. You are appreciated. I checked both your profiles and neither has any way to donate, so this is currently what I have the ability to do.

Thank you!

Hi, Still facing this CVE alert, are there any plan to fix and release next version?
 URL

@Syed-Shahul-Hameed more that I have to spend time on answering this question slower the release is. As soon as I find time -- this is a non-paid voluntary activity for me, alongside a real job and family -- I will make a release.

In the meantime I just hope users will eventually push back on security tool vendors that cause all of this stress and fake alarm: Security Circus at its worst.

I don't think that 'fake alarm' is appropriate. We were able to execute an DOS attack on a web application in a test system by using this attack vector.

Does the new 2.13.2.1 already contain the fix? Sorry, I've got to ask, as there are no release notes.

@cowtowncoder : kudos for all the work regarding this nasty thing!

You mentioned a probable fix for 2.12. What are your current thoughts/plans for that?

Just quoting cowtowncoder: URL

Leaving this here just in case someone hasn't noticed it: URL

@rebbuh Yes, 2.13.2.1 does contain the fix. Micro-patch release notes are bit tricky to find as they are on minor version page like so:

 URL

but I will update fixed-in version of this issue.

On 2.12.6.1: I will do that, it is only a question of finding time. Right now my time for OSS is limited and although getting this version out is the highest priority item, it has to compete with paying work and family and my personal health (I'm fine, not to worry -- and aim to stay that way! -- but it all takes work).

So please be patient: I will update this issue, Tweet, once version is out.

In the meantime, anyone able to is recommended to use 2.13.2.1 that is already out.

We were able to update to 2.13.2.1 at my company without an issue.

BlackDuck (OSS scanner tool) already recognizes that there is a version 2.13.2.1 and that it fixes the CVE, but the automatic scanner mistook it for being version 2.13.2, so we had to manually verify that we’re only using 2.13.2.1 and update the BlackDuck report to say that.

Taylor

@TaylorSMarks or others, sorry to ask a noob question, but the 2.13.2.1 does not seem to be in the Maven central just yet, how can we upgrade to that micro patch version ?

@bbarman4u it is on maven central. Only jackson-databind got an update though.

Sorry for being lost, I see only the version update up to 2.13.2 here in the Maven Central, when we declare it explicitly in the Maven pom.xml also it does not resolve.

 URL
image

@bbarman4u thank you. As to `jackson-bom`, for micro-patches its versioning differs because there is no full matching set of versions, so a timestamp-based fourth part is added: so in this case you can either:

1. Use `jackson-bom` version `2.13.2` but override version for `jackson-databind`
2. Use `jackson-bom` version `2.13.2.20220324` which does (1) essentially (everything else has version 2.13.2 but `jackson-databind` 2.13.2.1)

@cowtowncoder first, thanks a lot for your work! can i update only jackson-databind、jackson-core and jackson-annotations to 2.13.2(or 2.13.2.1), but let jackson-dataformat-xxx and jackson-module-xxx still 2.12.x. as 2.12.6.1 is still on the way, it is compatible to do this.

@zhanganyuan it depends on the module. Generally 2.12.x does not always work with 2.13.x. However sometimes it will work fine.

@zhanganyuan As general guidance, no, you should NOT mix different minor versions; exception being that in certain directions you may use "adjacent" ones. In particular: "lower" component like `jackson-core` MAY have HIGHER minor version than "higher level" component. So what is expected to work (but not guaranteed) would be:

* `jackson-core:2.13.x` but `jackson-databind:2.12.x`
* `jackson-databind:2.12.x` but `jackson-dataformat-xx:2.11.x`

but the reverse is not likely to work.

That said I would strongly recommend attempting to align all minor versions.

Quick note: I hope to release 2.12.6.1 today (2022-03-26) but there are some on-going discussions about use of `jackson-bom` (see URL that I think need to be resolved first. Stay tuned!

@cowtowncoder another thank you for your work on jackson-databind 2.13.2.1

Jackson 2.12.6.1 released with jackson-bom 2.12.6.20220326: on its way to Maven Central now.

If anyone knows how to submit an update to include exclusion for 2.12.6.1, that'd be great.
I don't know how to do that and will focus on getting 2.13.2.2 out today.

You can send an email to `cpe_dictionary@nist.gov` and ask them to amend the CPE list to reflect that 2.12.6.1 should not be affected.

I did that a few hours before I posted the above; they didn't respond.

I've at least submitted PR URL to improve CVE-2020-36518 with the 2.12 version fix.

Thank you @JoshDM. I think the omission is just due to update indicating 2.13.2.1 when 2.12.6.1 was not yet available.

@JoshDM & @cowtowncoder got you guys updated on the GHSA. Thanks for the contributions 👍

Finally found time to release:

* `jackson-databind:2.13.2.2` with corrected GMM manifest
* Optional `jackson-bom:2.13.2.20220328` bill-of-materials that includes version set that includes `jackson-databind:2.13.2.2` overrides over other `2.13.2` components

Please let me know if there are further issues; this SHOULD work with Gradle, similar to 2.12.6.1.

@blazespinnaker nobody is questioning that people parse untrusted json (most people do!), and I'm sure quite a few of those use the affected deserializer.

However, why do you think this is a DoS issue for those affected? The bug fixed here has the potential to throw a stack overflow exception, however that can be handled cleanly (and it *is* handled cleanly eg by app servers), so it is not at all clear how an attacker could cause DoS here. Just causing an error is not DoS, there's other ways to do that, you have to actually affect other users of the service.

@yawkat That's good context, however this is what I read at the top:

Perhaps that should be changed to reflect your perspective? I don't want to get into specifics, but I've done experimentation and the severity seems justified. I could be missing something though and would appreciate further techncial reassurance.

@cowtowncoder Agreed, the scanners are a pain because they are very dumb. I encourage folks to know their frameworks and use tools like opencve if they can.

@blazespinnaker Ah, good point on my bit misleading description: it gives impression that there is on-going memory retention which is... sort of not the case -- thread stack is exhausted, not heap, and the actual overload is bit different. And at the point of SOE memory retention will disappear.
This does not mean there is no possibility of DoS -- there is a way to cause excessive load with modest-sized documents -- but my description suggests a bigger problem.

Anyone have suggestions for a better description; maybe something to copy from an existing issue? :)

Wondering if there is a 2.13.3 version planned that includes fix for this issue?

To add some context to my question about 2.13.3 - our pom.xml files are structured in a way (with properties) that assigns the same property (version) to all jackson dependencies. The version is never a range, but a fixed version. That can of course be changed, but we prefer fixed versions aiming for repeatable build results. I think we're not the only organization which structures pom.xml files that way.

So changing just one dependency to a slightly different version like 2.13.2.2 is of course possible, but requires some work and will have to be rolled back as soon as 2.13.3 (or 2.14.0) comes out. Otherwise we have to remember to keep two properties in sync in pom.xml file which we want to avoid.

Then again, most important thing is that there is a fix for this CVE and I'd like to thank everybody who has contributed to that.

From NIST

Thank you for bringing this to our attention. We appreciate community input in order to provide the most accurate and up-to-date information as possible. After reviewing publicly available information, we have made the appropriate modifications to reflect that the affected versions are up to (excluding) 2.12.6.1 and from (including) 2.13.0 up to (excluding) 2.13.2.1. Please allow up to 24 hours for the changes to be reflected on the website and in the data feeds.

V/r,
Common Platform Enumeration Team
cpe_dictionary@nist.gov

Hi, is this a typo mistake? I find the bom version of 2.12 mentioned above is `2.12.6.1.20220326`, but in the release notes it's `2.12.6.20220326`.

image

Hi @cowtowncoder, any way to determine if one is affected or not?
You write that UntypedObjectDeserializer needs to be used in combination with Collection / Map - at least this is what I understand.
I scanned my code and we are not using UntypedObjectDeserializer, however, it might be used by other jackson-databind code that we use. Could you please clarify how to determine if one is affected by this CVE?

@cowtowncoder

use with jackson-bom version `2.12.6.1.20220326`

the version for the bom here looks incorrect, should be `2.12.6.20220326` ?

Hi @cowtowncoder, could you please clarify what methods are affected by this vulnerability? I am specifically interested if ObjectMapper.readTree is affected?

readTree is not affected, readValue is affected if you use untyped fields somewhere in the object graph youre deserializing

@MaximilianTews see #2816 for more information. This is just a rewrite of that fix.

And @yawkat is correct: `readTree()` is not affected. `readValue()` MAY be affected BUT ONLY if the nominal type is or includes target type of `java.lang.Object`. Term "untyped" in Jackson context means `java.lang.Object` as binding target type (either explicitly, or via raw types, `?` type parameter and so on).


Question: Pi-hole admin pages will load in an iFrame, but table data will not (Invalid JSON response)?

At this moment in time its:

root@222b35f401b9:/# lighttpd -v
lighttpd/1.4.53 (ssl) - a light and fast webserver

So you should be good to go

There are still a few things to look at around env vars before we can think about looking at this, but it is probably worth noting that `lighttpd` will be dropped in the future with Pi-hole 6.0 (no ETA) - so not 100% sure it's worth the work for this particular use case which has a perfectly adequate workaround in using `external.conf` for now

Pi-hole admin pages will load in an iFrame, but table data will not (Invalid JSON response)
### Versions

- Pi-hole: [v5.12.2]
- AdminLTE: [v5.18.1]
- FTL: [v5.15.1]

### Platform

- OS and version: Raspbian Buster

- Platform: Raspberry Pi 4B

### Expected behavior

Normal functionality in an Organizr iFrame after commenting out "X-Frame-Options" => "DENY" in lighttpd.conf on the Pi-hole host machine.

### Actual behavior / bug

Pi-hole AdminLTE pages will load, but table data will not. Also, if a password is enabled, one can never get past the login screen -- the login page just keeps reappearing. With the Pi-hole password disabled, any pages that contain table data generate an error of "DataTables warning: table id=groupsTable - Invalid JSON response.", or similar depending on which DataTable is applicable to a given AdminLTE page.

### Steps to reproduce

Steps to reproduce the behavior:

1. Go to /etc/lighttpd/lighttpd.conf and comment out "X-Frame-Options" => "DENY" (as shown below), remove Pi-hole password using pihole -a -p

# If the URL starts with /admin, it is the Web interface
$HTTP["url"] =~ "^/admin/" {
 # X-Pi-hole is a response header for debugging using curl -I
 # X-Frame-Options prevents clickjacking attacks and helps ensure your content is not embedded into other sites via , or .
 # X-XSS-Protection sets the configuration for the cross-site scripting filters built into most browsers. This is important because it tells the browser to block the response if a malicious script has been inserted from a user input.
 # X-Content-Type-Options stops a browser from trying to MIME-sniff the content type and forces it to stick with the declared content-type. This is important because the browser will only load external resources if their content-type matches what is expected, and not malicious hidden code.
 # Content-Security-Policy tells the browser where resources are allowed to be loaded and if itâ€™s allowed to parse/run inline styles or Javascript. This is important because it prevents content injection attacks, such as Cross Site Scripting (XSS).
 # X-Permitted-Cross-Domain-Policies is an XML document that grants a web client, such as Adobe Flash Player or Adobe Acrobat (though not necessarily limited to these), permission to handle data across domains.
 # Referrer-Policy allows control/restriction of the amount of information present in the referral header for links away from your pageâ€”the URL path or even if the header is sent at all.
 setenv.add-response-header = (
 "X-Pi-hole" => "The Pi-hole Web interface is working!",
# "X-Frame-Options" => "DENY",
 "X-XSS-Protection" => "1; mode=block",
 "X-Content-Type-Options" => "nosniff",
 "Content-Security-Policy" => "default-src 'self' 'unsafe-inline';",
 "X-Permitted-Cross-Domain-Policies" => "none",
 "Referrer-Policy" => "same-origin"

2. Configure Pi-hole admin pages to load through an Organizr iFrame tab
3. Click on 'Groups'
4. See error

## Debug Token

- URL:

 URL

### Screenshots

Screenshot 2022-09-27 082202

### Additional context

Organizr is such a great tool for organizing web-based setup pages for applications on one's LAN. Pi-hole isn't the only application that requires minor tweaks to work in an iFrame, but it would be nice to know what those tweaks are for full Pi-hole admin page functionality. Any help would be greatly appreciated!

This appears to only be an issue when Organizr and Pi-hole are installed on different hosts. When they're on the same host, and particularly if installed in Docker containers, the TableData JSON errors disappear and so does the problem with the perpetual login screen -- so there's no need to disable the password.

This leaves only the need to change the X-Frame-Options in a way that survives updates, which for Docker installations is addressed here:

 URL

This is a Feature Request.

We receive a high number of requests, but as a free open-source project run by volunteers we have very limited resources.
Please, we ask that you open your Feature Requests at our Discourse Forum.

Thank you for your understanding.

---

**Pi-hole links:**
- Discourse Forum: URL
- Documentation: URL
- Feature Requests: URL

Feature request to add frame options as docker env variable
This is a...

- [x] Request for a new or modified feature
- [ ] Issue trying to run the docker image
- [ ] Issue trying to build / test / develop the docker image

## Description
I would like to embed the pi.hole/admin interface in an iframe inside Home Assistant. But the "X-Frame-Options" => "DENY" inside lighttpd conf is preventing this.
I can't override it in external.conf as it won't allow a duplicate setting.

Can this be added as a env variable that replaces the setting during docker start?

Have you tried using a volume to overwrite the lighttpd conf?

Yes that would be one way to do it, but wouldn't it be nicer to be able to control it with a variable? That way any improvements to the main config would be kept on new releases without having to manually merge files.

I suppose it's not too difficult to add `sed -i '/X-Frame-Options/d' /etc/lighttpd/lighttpd.conf` controlled by an environment variable.

Until that happens, you could make a persistent customization that always runs on boot by adding the `sed` command to a bash script which is then volume mounted into the startup auto executed scripts folder. `-v :/etc/cont-init.d/10-my-prestart-script.sh`

So real work solution

1. Create a file `allow-iframe.sh` on host machine

shell
#!/usr/bin/with-contenv bash

# Remove header
sed -i '/X-Frame-Options/d' /etc/lighttpd/lighttpd.conf

# We stop it, watchdog will restart it with our conf
/etc/init.d/lighttpd stop

2. When you will start your docker image need to add a volume
`... -v :/etc/cont-init.d/21-allow-iframe.sh`
21 - because we have 20-start.sh which replace lighttpd.conf to new. And that is cause why need to restart the lighttpd service

file would be owned by my user as its part of the git pull to that device, I am running OpenMediaVault so essentially debian 9, I will try to chown file to root and see what happens at work rn.

@shanehughes1990

idk if it's just a typo or not, but take out the leading . on the line for the shell script and try that. you're not executing it from that line, you're simply copying it to the container (if I understand this correctly, i'm still pretty new to docker) and then it runs in order because of the 21. I setup this shell script in a config folder on the host and restarted the container using portainer, after adding this to the volume section and it worked a treat. make sure you've chmodded the script so that it's executable as well.

<img width="966" alt="Screen Shot 2019-10-10 at 10 23 53 AM" src=" URL

Seconding this, an environment variable for setting this would be great.

FYI, you can add this to your `external.conf` to overwrite the default setting:

$SERVER["socket"] == ":443" {
 $HTTP["url"] =~ "^/admin/" {
 # Allow framin (for my home-assistant)
 setenv.set-response-header += (
 "X-Frame-Options" => "Allow"
 )
 }
}

I have this with SSL. Should work if you change it to :80. I don't know lighttpd well enough to know of the $HTTP will work outside of the server block (didn't test it).

Add that to `external.conf` and mount it on start:

 volumes:
 - my-external.conf:/etc/lighthttpd/external.conf

It's a better solution than script using `sed`.

@guice as long as your answer is really great it doesn't work very well.
Current Pi-Hole runs on Lighttpd **1.4.45** and _setenv.set-response-header_ is supported since **1.4.46** (!). So close... _add-response-header_ on the other hand throws an error because there's a duplicate.

😬

Sorry. I'm using a local-install of Pihole on RasPi. I didn't even look at lighttpd versions. This is the method I'm using there, but admittedly, it's lighttpd 1.4.53.

I just connected dots from there: external.conf is meant for external configurations, and you can mount it directly via docker. Didn't even think the `set-response-header` wouldn't work within the docker version due to _one_ (minor) version number. Sorry. :(

No worries, it's not your fault. It's really good, clean, solution. It just needs some time before we can use it 🙃

In the meantime I think I need to fall back to mounting entire lighttpd.conf.

Hi, just dropping an idea. As this is a long-standing issue (2 years!), while the environment variables might take longer to be reworked, could the DENY value be changed to SAMEORIGIN, so that at least we can iframe it inside the same homeassistant docker site? Or is there any drawback to not use sameorigin either?
Thanks.

I did it, but it logs me off after loging in, any solution on that?

correct Docker-compose Volume Path is:
`volumes:
 - ./my-external.conf:/etc/lighttpd/external.conf
`

If you use Nginx to reverse proxy a quick fix would be to add this line to it's location block:

`proxy_hide_header X-Frame-Options;`

I'm leaving this here for anyone that finds this solution later. You need to mount the .conf file to /etc/lighttpd/conf-enabled as mentioned in release 23.01

tried the solution mentioned by the @pralor-bot and it actually fixes a part of the problem: the PiHole UI is displayed in the iframe now (inside the home assistand GUI).

However loggin in does not work - the login-screen will be displayed over and over again.

Does anyone has a suggestion ?


Question: Bug: view on open sauced location?

For first contributors to work on this you can do the next:
1. 🔍 Search for the keyword in the broken button (View On OpenSauced)
2. 🤔 Try to understand the component and code written within it.
3. 💭 Figure out where did we use this component Solution/spoiler 😄 The needed function: injectViewOnOpenSaucedButton
4. 👍 When you find the function related to this component, you can add extra logic to check whether it's the same logged-in user profile. (you can do this by finding a unique selector for user profile) Solution/spoiler 😄
You can do this by checking the Edit profile button, Solution/spoiler 😄
the related class selector: js-profile-editable-edit-button

@bdougie @Anush008 @diivi, what's your input about this? I am considering labeling it as a good-first-issue if it's clear and not blocking any business logic.

Bug: view on open sauced location
### Describe the bug

When you open a profile that does not have a bio, the view on open source will not show.

### Steps to reproduce

1. go to your profile
2. remove your bio
3. refresh the page :)

### Affected services

browser-extension

### Platforms

_No response_

### Browsers

_No response_

### Environment

_No response_

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

### Contributing Docs

- [ ] I agree to follow this project's Contribution Docs

This is related to: URL and URL

# Suggested solution:

1. 🔍 find a unique identifier for the profile menu, by doing the next:
2. 🕵 inspect the profile page, the bio and the edit button.
3. 👍 see if we can find a unique identifier to use it.
4. 🏥 review the code and the JS query selector to use that identifier

# Challenge

Let this work on your profile and other people's profiles.
For your profile, it will be under the bio information on top of the edit profile,
For other people's profiles

 😄 Hint

You might find a unique selector for the edit profile button
 Your own profile selector: show the selector

`btn-block js-profile-editable-edit-button`

 Other people profile selector:

`user-following-container`

 Use a condition maybe
You can check if the selector is not provided for your profile to trigger the selector for other people profiles, and add the button, in cases it will be need to be prepended.

@amartya3ac, the issue is assigned to you. Feel free to take issues to check the contributor guide as well:

 URL

@diivi I have implemented the solution can you tell me how can i raise a pr and what all should I include ?

Bug: odd location of view on open sauced
### Describe the bug

The location of view profile on open sacued is okay on other pages, but it's odd on personal page here is a screenshot

view of open sauced location

### Suggestions:

Add a control to check if it's the personal page to add it on the right location,

### Steps to reproduce

1. open your own profile page.

### Affected services

browser-extension

### Platforms

_No response_

### Browsers

_No response_

### Environment

_No response_

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

### Contributing Docs

- [ ] I agree to follow this project's Contribution Docs

Done, feel free to ask for help if you got stuck or needed any support, good luck 👍


Question: Reference to `this` is unknown inside cloud.Api method definition?

@Chriscbr

Since my technical knowledge here is very limited, it might be that this suggestion is really off, so please don't take me too seriously.

What if we take a slightly different approach and treat all global variables as Root members making:
1. Create a $FILE_ resource
2. Put all code inside the init function
3. transform all global `let`/`let var` statements into members
5. Instantiate $FILE_ resource

Making this:

ts (wing)
bring cloud;

let b = new cloud.Bucket();

let handler = inflight (payload: str) => {
 assert(b.list().length == 0);
 b.put("hello.txt", "world");
 assert(b.list().length == 1);
});
new cloud.Function(handler) as "test:put";

Into this:

ts (wing)
bring cloud;

resource $FILE_AppW {
 b: cloud.Bucket;
 handler: inflight (str);
 init(){
 this.b = new cloud.Bucket();
 this.handler = inflight (payload: str) => {
 assert(this.b.list().length == 0);
 this.b.put("hello.txt", "world");
 assert(this.b.list().length == 1);
 };
 new cloud.Function(this.handler) as "test:put";
 }

}
new $FILE_AppW();

The above code doesn't work because of #1966 (and maybe other stuff) and I don't see how it solves your `TypeAnnotation` problem... so I don't know how useful it is for you 🤷

Reference to `this` is unknown inside cloud.Api method definition
### Community Note

### I tried this:

`wing it`

ts
bring cloud;

class TaskList {
 init() {
 }

 inflight add(title: str): str {
 let id = "1";
 return id;
 }
}

class TaskListApi {
 api: cloud.Api;
 task_list: TaskList;

 init(task_list: TaskList) {
 this.task_list = task_list;
 this.api = new cloud.Api();

 this.api.post("/tasks", inflight (req: cloud.ApiRequest): cloud.ApiResponse => {
 let id = this.task_list.add("title");
 return cloud.ApiResponse { status:201, body: id };
 });
 }
}

let task_list = new TaskList();
let t = new TaskListApi(task_list);

Then in wing console I tried to `POST` to `/tasks` a simple request with `body: {"title": "Yossarian"}` and `content-type` header `application/json`.

### I expected this:

Wing console to print: success.

### Instead, this happened:

sh
evalmachine. :19
 async handle(req) { const { this } = this; {
 ^^^^

SyntaxError: Unexpected token 'this'
 at new Script (node:vm:100:7)
 at createScript (node:vm:265:10)
 at Object.runInContext (node:vm:296:10)
 at /Applications/Wing Console.app/Contents/Resources/app.asar/dist/vite/electron/main/index.js:238:149
 at new Promise ( )
 at n2t (/Applications/Wing Console.app/Contents/Resources/app.asar/dist/vite/electron/main/index.js:237:9364)
 at Object.activity (/Applications/Wing Console.app/Contents/Resources/app.asar/dist/vite/electron/main/index.js:237:8767)
 at Object.withTrace (/Applications/Wing Console.app/Contents/Resources/app.asar/dist/vite/electron/main/index.js:238:8269)
 at XH.invoke (/Applications/Wing Console.app/Contents/Resources/app.asar/dist/vite/electron/main/index.js:237:8689)
 at /Applications/Wing Console.app/Contents/Resources/app.asar/dist/vite/electron/main/index.js:237:2531

### Is there a workaround?

Yes, this seems to work (notice assignment of `self`):

ts
bring cloud;

class TaskList {
 init() {
 }

 inflight add(title: str): str {
 let id = "1";
 return id;
 }
}

class TaskListApi {
 api: cloud.Api;
 task_list: TaskList;

 init(task_list: TaskList) {
 this.task_list = task_list;
 this.api = new cloud.Api();

 let self = this;

 this.api.post("/tasks", inflight (req: cloud.ApiRequest): cloud.ApiResponse => {
 let id = self.task_list.add("title");
 return cloud.ApiResponse { status:201, body: id };
 });
 }
}

let task_list = new TaskList();
let t = new TaskListApi(task_list);

This code now runs into a different issue #1969

### Component:

Compiler

### Wing Version:

0.8.26

### Node.js Version:

19.3.0

### Platform(s):

MacOS

### Anything else?

_No response_

This is true for any inflight closure defined within a resource, not just API handlers.

@Chriscbr please keep this ticket in mind when working on #1448

Tested this today, and calling the API currently causes a runtime error when I call the API in Wing Console: "Unknown error: Missing environment variable: API_HANDLE_5f3a2e5e"

existing closure mechanism is causing permissive permissions
### Community Note

### Feature Spec
Generate custom resources from inflight closures

ts
let my_closure = (x: num) => {
 print("${x}");
};

We would like the compiler to generate a custom resource with a single method from this closure. Modeling closures as resources will allow us to correctly handle all use cases of closures and jsify it correctly.

### End consumer Impact

The way we calculate permissions is based on the resource (`this.*`) mechanism and this ticket is required in order to make sure that we do not give permissive permissions to resources when capturing them in this way:

ts (wing)
let b = new cloud.Bucket();
new cloud.Function(inflight () => {
 b.put("aa", "aaa");
});

will give all permissions for the bucket, while:

ts (wing)
bring cloud;
resource A impl cloud.IFunctionHandler {
 b: cloud.Bucket;
 init(b: cloud.Bucket) { this.b = b;}

 inflight handle() {
 this.b.put("aa", "aaa");
 }
}
new cloud.Function(new A(new cloud.Bucket()));

will give minimum permission

### Use Cases

For example #1447 for an inflight closure instead of a resource method will work out of the box once #1447 is fixed.

### Implementation Notes

_No response_

### Component

Compiler

@staycoolcall911 , I am turning this into P1 (see End consumer Impact section in description for details)

The latest update is I've run into issues issues when trying to extend the transformation further to automatically "close over" free variables. The problem I found is that in order to do so, we would need to produce `TypeAnnotation` nodes in our AST for each referenced variable:

js
bring "my_lib" as my_lib_alias;

let foo = new my_lib_alias.Foo();
let bar = foo.bar; // type: constructs.Construct

// before
let handler = inflight () => {
 log(bar.toString());
};

// after
resource $Resource1 {
 bar: constructs.Construct; // PROBLEM
 init(bar: constructs.Construct) {
 this.bar = bar;
 }
 inflight handle(req: str): str {
 log(this.bar.toString());
 }
}
let handler = new $Resource1(b);

At the line above labeled PROBLEM, we see that we need a type annotation for `constructs.Construct` (though in practice it could be any type really). The type checker has code for resolving `TypeAnnotation` into `Type`, but there isn't a safe way to convert arbitrary `Type` values into `TypeAnnotation` nodes without introducing new `bring` statements and so on. This would be pretty hairy.

So instead I'll try investigating how to solve #1447 so that the transformation doesn't need to close over any variables.

Curious... I guess if we take that a step further and transform the handler, we could get something like this:

js
bring cloud;

resource $FILE_AppW {
 b: cloud.Bucket;
 handler: inflight (str);
 init(){
 this.b = new cloud.Bucket();
 resource $Inflight1 {
 root: $FILE_AppW;
 init(root: $FILE_AppW) {
 this.root = root;
 }
 inflight handle(payload: str) {
 assert(this.root.b.list().length == 0);
 this.root.b.put("hello.txt", "world");
 assert(this.root.b.list().length == 1);
 }
 }
 this.handler = new $Inflight1(this); // is passing `this` logical?
 new cloud.Function(this.handler) as "test:put";
 }

}
new $FILE_AppW();

It does look like it could fix the problem I was describing with `Type`'s and `TypeAnnotation`'s since only one value needs to be passed into the constructor of `$Inflight1`, and the type of it is `$FILE_AppW` which we could calculate easily.

But I think the issue is by giving `$Inflight1` access to `this`, then all of the children resources' inflight clients also need to be bundled into the cloud function. So for example, suppose the user's app has other resources, like counters and queues and so on, and the particular inflight does not need to use them. Since `$Inflight1` has the entire app as a child resource (named `root`), it will need permissions for all the app's resources inflight, which defeats the purpose of least privilege permissions.

I think the idea of having the app as a single resource is pretty interesting though, I will definitely keep it in my back pocket


Question: d405 calibration issue?

You are very welcome. It's great to hear that you had a successful outcome. Thanks very much for the update!

d405 calibration issue
| | |
|---|---|
|**librealsense**|2.56.3 RELEASE|
|**OS**|Windows|

Intel RealSense Viewer / Depth Quality Tool has crashed with the following error message:

Unhandled exception escaping from a worker thread!
Error type: Access Violation!

Hi @muxiddin19 Please see URL for information about this error. Thanks!

Hi @muxiddin19 Do you require further assistance with this case, please? Thanks!

on chip calibration error persist
| | |
|---|---|
|**librealsense**|2.56.3 RELEASE|
|**OS**|Windows|

Intel RealSense Viewer / Depth Quality Tool has crashed with the following error message:

Unhandled exception escaping from a worker thread!
Error type: Access Violation!

Hi @Sangeethaj152 Please first check whether permission to access the camera hardware and permission for applications to access the camera are both set to 'On' in the Windows camera privacy settings. This can be done using the instructions at URL

If you are able to enable the depth stream in the RealSense Viewer tool then an alternative to performing on-chip calibration to improve the calibration is to reset the camera in the Viewer to its factory-new default calibration using instructions at URL

Case closed due to solution achieved and no further comments received.


Question: Cursor has problems in Wayland?

Are you running the latest master branch. We very recently (<2 weeks) applied a patch on pointer constraints.

If running the latest doesn't solve this, could you try `sway` and/or `weston` before we start looking too deep.

Cursor has problems in Wayland
Hi,

I don't know if it's related to RetroArch apparently lacking pointer confinement on Wayland, as I commented here:
 URL

...But for example, Lemmings or Ishar have major problems with cursor movement on Wayland, like cursor is stuck on the right half of the screen, or in a certain area.
This happens on all Wayland compositors I tried: Weston, Sway and tinywl.

Has anybody tried this core on Wayland compositors?

Do other cores that read mouse input from RA behave the same? (Like PUAE.)

@realnc No, PUAE works perfectly well on that regard, only dosbox-core seems to be affected.

@realnc Also, I have been playing around with all core-specific options related to mouse handling, and they make no difference: mouse seems to be trapped inside certain regions.
Do you have a Wayland system you can test this on?

I have no idea what to do about it. The core just reads mouse input from RA. The cursor behaves in whatever way RA tells the core to behave.

In Ishar 1, RA traps the mouse inside a small window in the center of the screen.

@realnc I have been thinking about this bug: Is there a place where you initialize the screen area size for the mouse? I think that could be missing, because the cursor works perfectly once it's moved around the screen.

There is no such thing. The core has no idea how big the screen is. The core simply tells RetroArch what the game's resolution is (like 320x200) and RA sends mouse input suitable for that resolution. In X11, if you don't enable "grab mouse" in RA (or "game focus" which also grabs the mouse ), you get the same issue where you need to move the mouse in all directions first. "Grab mouse" is a RA hotkey. I have it bound to F11.

Maybe "grab mouse" doesn't work correctly on Wayland?

@realnc Yep, that has to be it. These features must be implemented by the compositor (labwc in this case) so I guess it's missing. I will ask the labwc devs :)

@realnc It seems that RetroArch is missing mouse grab in the Wayland backend:

 URL

So, that's the problem it seems.

Missing mouse grabbing feature?
Hi there!

I have noticed that, with the labwc compositor, some RetroArch cores have mouse problems.
For example, dosbox-core has pointer wrongly locked to a small area of the screen until the cursor is moved around beyond this small area:
 URL

This seems to be caused by the lack of mouse grabbing, which is a feature implemented (I guess) by the Wayland compositors.

So, would it be possible to have this feature implemented, please? Or maybe there's a problem with the current implementation?

Thanks!

"Mouse grabbing" in the wayland context is two things AFAIU:
- cursor confinement which restricts the cursor to a specific area of the surface
- cursor confinement which locks the cursor into a specific place

Related to the lock confinement, there is also a relative pointer motion protocol which (even though the actual cursor doesn't move) tells the wayland client about relative movements of the cursor and thus can be used by applications to render their own cursor. So I think the compositor side of the puzzle should be working, maybe RetroArch / dosbox-core (I don't really know how either of them work) just don't implement the "grab cursor" feature when it detects that its running under wayland?

@Consolatis I would like, at last, know where the problem really is: on the Wayland/labwc or on the RetroArch side. Thanks for trying to help me finding out! :)

What function should be used from RetroArch's side to grab the mouse, please? I can look on the RetroArch sources for that function, so I can find out if it's being used or not.

The idea of the missing compensation for scaling is also very plausible: the games in RetroArch are scaled from, let's say, 320x200, to 1920x10080.
Would the lack of scaling compensation cause cursor to be confined to a region and then moving it would finally "unlock" it?

Y es, I'm running the latest GIT code: cursor confinement in fullscreen mode works perfectly in Scummvm since these changes you mention! scummvm runs on SDL2.

Sway and Weston have the exact same problem with the dosbox-core core in retroarch. I understand they implement all the grabbing functions?

I'd assume so, yes (but didn't actually verify).

I don't think so, that sounds like some retroarch / dosbox issue.
I've just tested using an output scale of `2` for a nested output and the confinement seems to work correctly in that case as well.

Should be either `zwp_pointer_constraints_v1_confine_pointer` or `zwp_pointer_constraints_v1_lock_pointer`

@Consolatis Ah, thanks for taking a look :)
I am trying to implement mouse grab in RetroArch, exactly in `input_wl_grab_mouse()` where you pointed me to.

However, first problem is that never did any Wayland specific coding, so I am getting `undefined reference` when adding a call to `zwp_pointer_constraints_v1_confine_pointer` or `zwp_pointer_constraints_v1_lock_pointer`.

I did a grep search on `/usr/local/include` on my system (which runs latest stable WLRoots and the fantastic latest GIT Labwc on top of it with no problems at all), and had zero results, which is very puzzling.
Are these plain functions? Then I guess they should be on some system-installed header file under /usr... unless I am fundamentally wrong on something.

@Consolatis Ok, opened the issue in RetroArch: URL

Now, let's see what they say about this...

@Consolatis The last problem was on RetroArch and is now fixed!
 URL

So this can be closed.

Labwc is now my daily driver! Thanks a lot! This compositor is incredible.

Nice work. Congratz on adding the feature and getting it merged :)


Question: Data Wrangler view does not remember hidden side bar / bottom panel?

Hi folks! Thank you very much for your patience and for providing great feedback on this issue.

We've just added a setting to disable the automatic panel behaviour in pre-release version 1.13.3. The following setting needs to be disabled to prevent the panels from appearing on startup and on tab focus:
Image

Please feel free to give this a try and let us know if you have any additional feedback. Thanks!

Data Wrangler view does not remember hidden side bar / bottom panel
Type: Bug

# Behaviour

## Expected vs. Actual

When I hide the side bar and the panel in the Data Wrangler view, change the tab to an open notebook and return to the Data Wrangler view, the side bar and the panel show up again. I expect, them to stay hidden upon tab switch.

# Diagnostic data

- Jupyter extension version: 2023.10.1100000000
- Python extension version: 2023.20.0
- .NET Install Tool for Extension Authors extension version: 2.0.0
- Python package dependencies:

json
{
 "satisfied": {
 "pandas": {
 "installedVersion": "2.1.3",
 "requiredVersion": "0.25.2"
 }
 },
 "unsatisfied": {}
}

Extension version: 0.20.0
VS Code version: Code 1.84.2 (1a5daa3a0231a0fbba4f14db7ec463cf99d7768e, 2023-11-09T10:50:47.800Z)
OS version: Linux x64 6.5.9-1-MANJARO
Modes:

Hi @tea-de-kay, thanks for opening this issue!

Note that this behaviour is currently by design due to limitations with VS Code API. However, we are experimenting with some fixes to improve this experience. I will update this thread when the changes are checked in.

Wanted to add my support for this request. Often I will be switching between the data wrangler view & my jupyter notebook and it is frustrating to have to close the bottom and side panel. Awesome extension though, thanks for the extension!

Adding support as well. Definitely wish there were a way to have the panel and side bar auto close when exiting the data wrangler editor.

Adding support for this as well. This would be a great addition!

Hi folks! Just wanted to let you know that there is now an option to not automatically reveal the panels (both on startup and on tab switch) in pre-release version 1.13.3.

For more details, please see: URL

Will close this issue for now since this setting should address the pain points mentioned here, but please feel free to reopen if you have additional feedback. Thanks!

feature request: add option to disable opening sidebar+console
The current behavior of data-wranggler provides a poor UX on smaller screens. Please give an option to disable opening the side-bar and console view. Thanks!

Hi @bobcolner, thanks for making this feature request! This is a great callout - we're currently exploring different ways to address the panel real estate issue and will provide any updates here.

I would really appreciate this as well, since I find it incredibly annoying to have to close it again and again every time I just want to have a look at my data and don't need to see the basic Data Summary.

I just think an option should be added to not open the panel. I don't think it's a matter of re-thinking the side bar philosophy.

I was about to open an issue on this, then I realized it was already done. Please change this standard behaviour: it does not happen for other extensions!

Please add this, has been a pain for a while. Much appreciated!

It would be great to be able to change Data Summary panel behavior in Settings similar to enabling/disabling Column Insight.

Finally! :) Thanks a lot. I can confirm it is indeed working on the pre-release version


Question: 🗂 - Deprecated -moz-any and -webkit-any pseudo-classes used in Taiga UI CSS?

Closing this issue because ownership of Critters has moved to the Nuxt team, who will be maintaining the project going forward. This repo is archived and won't receive any future updates.

If the issue is still relevant, please re-post it to the actively-maintained fork at URL

🗂 - Deprecated -moz-any and -webkit-any pseudo-classes used in Taiga UI CSS
### Description

Dear Taiga UI Team,

I've encountered warnings during the build process of my Angular 19 or from version 17+ applications that uses Taiga UI. These warnings indicate the use of deprecated -moz-any and -webkit-any pseudo-classes in Taiga UI's CSS.

During the Angular build process (using ng build), the following warnings are generated:

[WARNING] 391 rules skipped due to selector errors:
[tuiAppearance][data-appearance=accent]:-webkit-any(a,button,select,textarea,input,label):not([data-state]) -> Unknown pseudo-class :-webkit-any
[tuiAppearance][data-appearance=accent]:-moz-any(a,button,select,textarea,input,label):not([data-state]) -> Unknown pseudo-class :-moz-any
... (many similar warnings for different appearances)....

Description of the issue:

These warnings stem from the use of -webkit-any and -moz-any, which are non-standard and deprecated. The standard replacement is the :is() pseudo-class.

Expected behavior:

Taiga UI's CSS should use the standard :is() pseudo-class to ensure compatibility with modern browsers and avoid build warnings.

Current behavior:

The build process generates warnings due to the presence of -webkit-any and -moz-any in Taiga UI's CSS. While the application may still function, these warnings clutter the build output and indicate the use of outdated CSS.

This is an Angular bug, Taiga UI has to use these classes to support Safari 13, which is the minimum required version for Taiga 4

Is there a way, where I can restrict or get rid of the warnings when am building, my bosses are not happy with warnings in builds but I successfully convinced them to start using Taiga UI. Any help would be appreciated Lol 🙏

I believe you can disable critical CSS inline as mentioned in original Angular CLI issue:
 URL

That issue was then closed as Critters issue:
 URL

Which was then discontinued – I suppose it needs to be fixed under this new fork that Angular uses:
 URL

Filed it just in case, but I wouldn't hold my breath since even though it's a valid selector – it's pretty much outdated for anything other than Safari 13 and below:
 URL

Let's see if this combination of selector helps, since they also cover all the browsers we must support

Thank you so much sir 🚀, time to update. Am a big fan of your Library. Much love for the changes ✅

Building Angular 17.1 with Angular CLI when using Bootstrap 5.3.2 results in Critical CSS Inlining failing because 2 rules are skipped due to selector errors
The 2 rules are the same rule listed twice:

- Generating index html...2 rules skipped due to selector errors:
 .form-floating>~label -> Did not expect successive traversals.
 .form-floating>~label -> Did not expect successive traversals.

`>~` is obviously an invalid selector, however it doesn't look like that's ever actually used in the Bootstrap code. I've checked myself, and one of the bootstrap devs has also confirmed it twbs/bootstrap#39323. The base style is simply `.form-floating > label` while there are a number of styles in the format of `.form-floating > [another selector] ~ label`. It seems one of those intermediary selectors is being removed during CSS processing.

There is a workaround of disabled Inlining of Critical CSS, but of course we're then missing out on a possible performance improvement.

This was reported as an Issue with Angular CLI last year, but they indicated it is a problem with Critters: URL

Even in the empty angular 17.1.0 project I get some errors

Screenshot 2024-01-26 232318

Chipping in from the ng-bootstrap repo, where there was also a bug filed last year about this problem and it was resolved intermittently by critters version `0.0.18`.
As it now resurfaces this likely is a regression.
I saw the warnings with `@angular-devkit/build-angular:17.2.1` which references critters at version `0.0.20`,

While the error looks similar, this issue is **not** a duplicate of #103. Because the whole project was heavily refactored in #124, specifically to solve #103. So it is unlikely to be the same root cause, and this issue needs its own proper investigation.

Same on my side. Setup an Angular ssr project from scratch including bootstrap and get exactly the same output running ng build.

Uploading warnings.png…

I see a lot of similar errors with picocss in Angular 17.2. For the moment I set "inlineCritical": false to have a working build.

We're adding SSR to our legacy Angular project after upgrading to Angular v17.3.6, and it's now crashing the server-side node app due to looping selector errors (both with Vite dev server and express production app builds). The CSS styles are Telerik's, which may be hard to mitigate:

Angular is running in development mode.
7 rules skipped due to selector errors:
 .k-input-label:dir(rtl) -> Unknown pseudo-class :dir

Besides the "inlineCritical" workaround, which doesn't seem(?) to work initially, I saw that the Angular team commented on maybe finding a temporary solution that involves pinning a working Critters version... can anyone here indicate what version that should be, as well as whether this is targeted for a near-term fix?

Here's a case where it fails on a valid, albeit deprecated selector that we have to use because we need Safari 13 (only Safari 14 supports `:is( )`, we have to use `-webkit-any( )`:
 URL

I'm getting same warning after upgrade my angular 17.3.0 to 18.0.6


Question: Pluto.TV Addon: DAZN Channels could no longer played?

oh wow. i dont get that all.
Is that a cloudflare message or their site returning it?

Pluto.TV Addon: DAZN Channels could no longer played
Hi @matthuisman , I noticed in the Pluto.TV addon that the DAZN (for examle "DAZN Darts x Pluto TV" or "DAZN International Football x Pluto TV") channels could no longer be played. The error message “Unable to play” always appears. Do you have an idea why that is?

kodi_slyg_pluto_dazn.log

Greetings Weri

Yes @kyl416 , I saw #657 . This entry has now been closed with the result that everything is working again. This is also the case, with the exception of the DAZN channels. Maybe @matthuisman no longer wants to support this because of the topic described?

Greetings Weri

thats not the case. the dazn channels are there. its just they need to switch to v2 hls manifest

Ah ok @matthuisman , where can this setting be made? Or does your addon need to be adapted for this?

Greetings Weri

Hi @matthuisman I noticed that after a certain time the stream from DAZN Darts x PlutoTV simply stops and ends up back in the Kodi main menu. It is irrelevant which Kodi version is used. Is this due to the addon or Inputstream.adaptive?
Greetings Weri
kodi_plutlo_omega.log

Hi @matthuisman I have created an additional debug.log for the problem presented.

kodi.log
Greetings Weri

itll be the ad breaks / x-discontuity. So inputstream.adaptive needs to support it fully
nothing i can do my end sorry

PlutoTV playlists / add-on OFFLINE until further notice
News article about what and why:
 URL

Have taken offline to keep Vultr happy and hopefully not take down my hosting.

in the meantime users could try URL instead

I'll search for alternatives and keep everyone posted here

Is the software you use to generate the m3u's for pluto open source, similar to maddox/pluto-to-channels?

+1 for this repo. Spun-up a container, locked network for jellyfin access only and working great. It will return the streams for my region only though (UY). I havent found a way to pull up brazil regions that I want.

ill try get a docker solution sorted so users can simply generate their own

got this:
"
Sorry, you have been blocked
You are unable to access nocords.xyz
"
;<

seems from cloudflare (can't upload screen capture to show the screen)

Yeah was getting that yesterday but the site loads using VPN. Not sure how viable that is long-term.

Hoping Matt comes up with a good solution in the next days/weeks..

Hello,
We are in a bad situation. But is there a problem if you generate xml epgs? The url you gave doesn't support french epg and others.
Thanks

i've also hacked together a node script that generates the m3u8 and xml for different regions (to get other regions, an IP address from that region is needed).

 URL

I hope this helps others.

Hey cool tool it actually helped me get the MX m3u8 and xml.

Hello, Thanks for these great work.

Is it possible to use same tvg-id as the one used by Matt.

Example:
- with your script every tvg-id are followed by the country code: 64edf6eaa7ec0d000812f58c-uk
- with Matt script the tvg-id is just 64edf6eaa7ec0d000812f58c

Thanks

Previously I found that pluto TV sometimes uses the same channel id for different channels in different locations. My local setup merges several playlists and xml files into one, and I wanted to ensure the ids were globally unique.

If you want, you could edit lib/plutotv/api.js and remove the '-${region}' suffixes from the ids in the m3u8 and xmltv functions.

Ok I understand your choice. I never detected the same if for different channels but you yes, so your choice is better.

I thought i was giving you feedback of issues by sending you a screenshot
stupid me

On Fri, 12 Jan 2024 at 12:13, Matt Huisman ***@***.***> wrote:

Thanks Matt. It's a shame Torrentfreak brought more attention to your work. Thanks for sharing a backup.

Try different VPN servers. US ones generally work for me.

It's a good alternative but far FAR inferior to what Matt had in terms of quantity and EPGs. Power to him. Hope he finds a way to bring it back..

Just use the 4v3ngR software and you will have exactly the same files as the one generated by Matt. I'm using it still few days and I 've detected no problem.

Great, see if someone package it into a docker solution so that everyone can install their own version e.g. on Vercel

Currently a noob and no idea how to use it but I will definitely try to learn :)

More than once a day. At least 3 times I think. So that's why you have to implement the tool on your system.

For those that are interested, I've pushed the scraper to the npmjs repo. There's no longer any need to check out the source code and do npm install etc. Do note that both npm and node (version >= 14) is required. Once node is installed on your system, a quick `npm install -g plutotv-scraper` is all that is needed to run it. There's also exttra options to support features other wanted.

running `plutotv-scraper --help` will provide some details of the options.

I'll also note that there is a "feature" where it will load config.json from the current directory. I'm not yet sure if this is a bug or a feature. For now it is a feature.

I also changed the license from AGPL to MIT, so everyone is free to use it however they want.

Finally, thanks to Matt for providing the original streams for so many years.

I read many posts, but I can't understand why can't someone upload another working Pluto addon

The playlists and epgs for Plex, Roku and Samsung now appear to have vanished

The Australian and NZ playlists and epgs have gone as well .
It looks to me like this is affecting all lists where Matt had his own urls redirecting to the official ones.
The Stirr list is still up (for the moment), but that use the official urls.

I wonder if a return to using the official urls without redirection might be acceptable to the providers/host?

Thanks for update. Please update here as well. I don't follow Twitter 👍

Awesome! Huge thanks! Is there an issue with Pluto that is different from the one highlighted earlier?